2022-11-23T00:59:10.0103111Z Requested labels: linux.2xlarge
2022-11-23T00:59:10.0103207Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T00:59:10.0103231Z Waiting for a runner to pick up this job...
2022-11-23T00:59:10.3042682Z Job is about to start running on the runner: i-038a48515f8572324 (organization)
2022-11-23T00:59:13.5410744Z Current runner version: '2.299.1'
2022-11-23T00:59:13.5416017Z Runner name: 'i-038a48515f8572324'
2022-11-23T00:59:13.5416421Z Runner group name: 'Default'
2022-11-23T00:59:13.5416852Z Machine name: 'ip-10-0-6-115'
2022-11-23T00:59:13.5418455Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T00:59:13.5418908Z Actions: write
2022-11-23T00:59:13.5419142Z Checks: write
2022-11-23T00:59:13.5419409Z Contents: write
2022-11-23T00:59:13.5419662Z Deployments: write
2022-11-23T00:59:13.5419906Z Discussions: write
2022-11-23T00:59:13.5420097Z Issues: write
2022-11-23T00:59:13.5420349Z Metadata: read
2022-11-23T00:59:13.5420591Z Packages: write
2022-11-23T00:59:13.5420786Z Pages: write
2022-11-23T00:59:13.5421023Z PullRequests: write
2022-11-23T00:59:13.5421311Z RepositoryProjects: write
2022-11-23T00:59:13.5421536Z SecurityEvents: write
2022-11-23T00:59:13.5421807Z Statuses: write
2022-11-23T00:59:13.5422048Z ##[endgroup]
2022-11-23T00:59:13.5424920Z Secret source: Actions
2022-11-23T00:59:13.5425569Z Prepare workflow directory
2022-11-23T00:59:13.7700361Z Prepare all required actions
2022-11-23T00:59:13.7859418Z Getting action download info
2022-11-23T00:59:14.2262654Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T00:59:14.5137088Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T00:59:17.0121790Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T00:59:17.2753067Z Getting action download info
2022-11-23T00:59:17.6745438Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T00:59:17.8573769Z Getting action download info
2022-11-23T00:59:18.2343811Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T00:59:18.3761674Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T00:59:18.3763244Z ##[group] Inputs
2022-11-23T00:59:18.3763506Z   build-environment: linux-focal-py3.7-gcc7
2022-11-23T00:59:18.3764299Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "docs_test", shard: 1, num_shards: 1,  runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
]}

2022-11-23T00:59:18.3765136Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T00:59:18.3765461Z   sync-tag: 
2022-11-23T00:59:18.3766115Z   timeout-minutes: 240
2022-11-23T00:59:18.3766317Z ##[endgroup]
2022-11-23T00:59:18.3766753Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 1, 2, linux.2xlarge)
2022-11-23T00:59:18.4462557Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T00:59:18.4462832Z with:
2022-11-23T00:59:18.4463320Z   github-secret: ***
2022-11-23T00:59:18.4463526Z   activate-with-label: false
2022-11-23T00:59:18.4463722Z   label: with-ssh
2022-11-23T00:59:18.4463899Z   remove-existing-keys: true
2022-11-23T00:59:18.4464086Z env:
2022-11-23T00:59:18.4464260Z   GIT_DEFAULT_BRANCH: master
2022-11-23T00:59:18.4464434Z ##[endgroup]
2022-11-23T00:59:18.5230401Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T00:59:18.5401192Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T00:59:18.5401454Z with:
2022-11-23T00:59:18.5401619Z   submodules: recursive
2022-11-23T00:59:18.5401817Z   fetch-depth: 0
2022-11-23T00:59:18.5402105Z env:
2022-11-23T00:59:18.5402263Z   GIT_DEFAULT_BRANCH: master
2022-11-23T00:59:18.5402449Z ##[endgroup]
2022-11-23T00:59:18.5603957Z ##[group]Run retry () {
2022-11-23T00:59:18.5604209Z retry () {
2022-11-23T00:59:18.5604426Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T00:59:18.5604638Z }
2022-11-23T00:59:18.5604823Z echo "${GITHUB_WORKSPACE}"
2022-11-23T00:59:18.5605040Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T00:59:18.5605251Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T00:59:18.5605453Z else
2022-11-23T00:59:18.5605646Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T00:59:18.5605825Z fi
2022-11-23T00:59:18.5606040Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T00:59:18.5621353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T00:59:18.5621582Z env:
2022-11-23T00:59:18.5621764Z   GIT_DEFAULT_BRANCH: master
2022-11-23T00:59:18.5621954Z   NO_SUDO: 
2022-11-23T00:59:18.5622112Z ##[endgroup]
2022-11-23T00:59:18.5713113Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T00:59:20.6612903Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T00:59:20.6613149Z with:
2022-11-23T00:59:20.6613341Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T00:59:20.6613530Z   fetch-depth: 0
2022-11-23T00:59:20.6613708Z   submodules: recursive
2022-11-23T00:59:20.6613892Z   quiet-checkout: true
2022-11-23T00:59:20.6614075Z   repository: pytorch/pytorch
2022-11-23T00:59:20.6614365Z   token: ***
2022-11-23T00:59:20.6614532Z   ssh-strict: true
2022-11-23T00:59:20.6614708Z   persist-credentials: true
2022-11-23T00:59:20.6615113Z   clean: true
2022-11-23T00:59:20.6615282Z   lfs: false
2022-11-23T00:59:20.6615448Z   set-safe-directory: true
2022-11-23T00:59:20.6615626Z env:
2022-11-23T00:59:20.6615792Z   GIT_DEFAULT_BRANCH: master
2022-11-23T00:59:20.6615962Z ##[endgroup]
2022-11-23T00:59:20.7694708Z Syncing repository: pytorch/pytorch
2022-11-23T00:59:20.7696247Z ##[group]Getting Git version info
2022-11-23T00:59:20.7696674Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T00:59:20.7697099Z [command]/usr/bin/git version
2022-11-23T00:59:20.7697344Z git version 2.37.1
2022-11-23T00:59:20.7699216Z ##[endgroup]
2022-11-23T00:59:20.7718443Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/01848a73-18bf-418a-888e-21eca9374e34' before making global git config changes
2022-11-23T00:59:20.7719125Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T00:59:20.7722345Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T00:59:20.7761105Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T00:59:20.7765995Z ##[group]Initializing the repository
2022-11-23T00:59:20.7768962Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T00:59:20.7798386Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T00:59:20.7798935Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T00:59:20.7799243Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T00:59:20.7799492Z hint: 
2022-11-23T00:59:20.7799773Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T00:59:20.7800121Z hint: 
2022-11-23T00:59:20.7800398Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T00:59:20.7800768Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T00:59:20.7801000Z hint: 
2022-11-23T00:59:20.7801384Z hint: 	git branch -m <name>
2022-11-23T00:59:20.7802019Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T00:59:20.7808000Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T00:59:20.7837079Z ##[endgroup]
2022-11-23T00:59:20.7837430Z ##[group]Disabling automatic garbage collection
2022-11-23T00:59:20.7840030Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T00:59:20.7866043Z ##[endgroup]
2022-11-23T00:59:20.7866376Z ##[group]Setting up auth
2022-11-23T00:59:20.7872293Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T00:59:20.7901180Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T00:59:20.8139660Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T00:59:20.8166479Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T00:59:20.8412781Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T00:59:20.8456217Z ##[endgroup]
2022-11-23T00:59:20.8456569Z ##[group]Fetching the repository
2022-11-23T00:59:20.8462042Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:00:09.9121754Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:00:09.9150213Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:00:09.9156698Z ##[endgroup]
2022-11-23T01:00:09.9157061Z ##[group]Determining the checkout info
2022-11-23T01:00:09.9158602Z ##[endgroup]
2022-11-23T01:00:09.9158938Z ##[group]Checking out the ref
2022-11-23T01:00:09.9163409Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:00:11.1917502Z ##[endgroup]
2022-11-23T01:00:11.1917973Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:00:11.1924767Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:00:11.1972045Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:00:11.2000079Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:00:11.2028317Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:00:11.2053961Z ##[endgroup]
2022-11-23T01:00:11.2054364Z ##[group]Fetching submodules
2022-11-23T01:00:11.2057801Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:00:11.2323178Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:00:11.2574387Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:00:11.2575276Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:00:11.2577739Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:00:11.2579764Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:00:11.2581985Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:00:11.2584352Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:00:11.2586782Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:00:11.2589103Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:00:11.2591630Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:00:11.2594396Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:00:11.2597055Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:00:11.2599940Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:00:11.2602942Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:00:11.2605963Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:00:11.2609162Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:00:11.2612428Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:00:11.2617484Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:00:11.2620732Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:00:11.2624234Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:00:11.2628023Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:00:11.2631824Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:00:11.2635820Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:00:11.2639830Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:00:11.2643854Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:00:11.2648035Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:00:11.2652378Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:00:11.2656913Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:00:11.2661476Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:00:11.2666032Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:00:11.2670747Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:00:11.2675349Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:00:11.2680123Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:00:11.2685128Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:00:11.2690122Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:00:11.2696869Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:00:11.2701975Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:00:11.2707297Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:00:11.2712670Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:00:11.2718217Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:00:11.2723989Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:00:11.2729721Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:00:11.2753007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:00:11.6148876Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:00:11.8303613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:00:12.0744573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:00:12.3613076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:00:12.6631100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:00:14.6379946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:00:20.7102341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:00:21.1169036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:00:21.6336957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:00:23.0624518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:00:24.1378521Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:00:25.5333276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:00:30.5955144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:00:31.3263725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:00:32.6636903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:00:33.8753779Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:00:34.0901230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:00:34.5458671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:00:34.8727660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:00:35.8840316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:00:36.3128878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:00:36.5079294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:00:36.7851100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:00:38.1085450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:00:38.5288706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:00:39.0301259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:00:44.9061619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:00:46.5836392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:00:47.0197008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:00:47.2820828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:00:52.7815593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:00:53.0092025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:00:53.2906241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:00:54.1208355Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:00:54.3281914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:00:54.6451503Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:00:54.9433582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:00:55.6168398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:00:57.6855774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:00:58.1609505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:01:00.3246442Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:01:00.3343703Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:01:00.3418011Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:01:00.3621154Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:01:00.3822939Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:01:00.4144674Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:01:00.9636348Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:01:00.9825719Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:01:01.0732363Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:01:01.1043556Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:01:01.3674284Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:01:01.7427538Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:01:01.9647034Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:01:02.0064803Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:01:02.0079181Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:02.0081228Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:02.0084503Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:02.0087548Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:02.0109526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:01:03.0091199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:01:03.5759292Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:01:04.6282690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:01:04.9717142Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:01:05.0628843Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:01:05.1170035Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:01:05.1264829Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:01:05.2111765Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:01:05.2442213Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:01:05.2525254Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:01:05.2891396Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:01:05.3106788Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:01:05.3531299Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:01:05.3637960Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:01:05.3651360Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:05.3672264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:01:13.2303487Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:01:13.2319234Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:13.2342603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:01:21.1521262Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:01:21.1616352Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:01:21.1746698Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:01:21.2597132Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:01:21.2611600Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:21.2614023Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:21.2637115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:01:22.3973429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:01:23.4779293Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:01:23.5277222Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:01:23.5464117Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:01:23.5590415Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:01:23.6570670Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:01:23.8787133Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:01:23.8812384Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:23.8814380Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:23.8837266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:01:24.2656304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:01:25.1119451Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:01:25.1404021Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:01:25.1541659Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:01:25.1553978Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:25.1574474Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:01:26.8610664Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:01:26.8628947Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:26.8630882Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:26.8653868Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:01:27.2982971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:01:28.1182767Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:01:28.1775112Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:01:28.1787230Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:28.1808726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:01:28.3982066Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:01:28.4066132Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:01:28.6389016Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:01:28.6407218Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:28.6409399Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:28.6432190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:01:29.0322000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:01:30.1233643Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:01:30.1855621Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:01:30.1932182Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:01:30.2028452Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:01:30.2331597Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:01:30.2410448Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:01:30.2670755Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:01:30.2755744Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:01:30.3155371Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:01:30.4134132Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:01:30.4371146Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:01:30.4384550Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:30.4386344Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:30.4388503Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:30.4390807Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:30.4412570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:01:31.4731106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:01:31.7506966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:01:32.9861796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:01:33.8744008Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:01:33.8876846Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:01:33.9467905Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:01:33.9720415Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:01:33.9733961Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:33.9755854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:01:34.3356661Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:01:34.4567139Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:01:34.4600899Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:01:34.4861735Z Entering 'android/libs/fbjni'
2022-11-23T01:01:34.4902176Z Entering 'third_party/FP16'
2022-11-23T01:01:34.4937439Z Entering 'third_party/FXdiv'
2022-11-23T01:01:34.4971742Z Entering 'third_party/NNPACK'
2022-11-23T01:01:34.5005960Z Entering 'third_party/QNNPACK'
2022-11-23T01:01:34.5041899Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:01:34.5076908Z Entering 'third_party/XNNPACK'
2022-11-23T01:01:34.5121708Z Entering 'third_party/benchmark'
2022-11-23T01:01:34.5155728Z Entering 'third_party/cpuinfo'
2022-11-23T01:01:34.5191860Z Entering 'third_party/cub'
2022-11-23T01:01:34.5226208Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:01:34.5266089Z Entering 'third_party/cutlass'
2022-11-23T01:01:34.5307113Z Entering 'third_party/eigen'
2022-11-23T01:01:34.5344453Z Entering 'third_party/fbgemm'
2022-11-23T01:01:34.5379683Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:34.5413196Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:34.5448447Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:34.5482744Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:34.5518092Z Entering 'third_party/flatbuffers'
2022-11-23T01:01:34.5555327Z Entering 'third_party/fmt'
2022-11-23T01:01:34.5591914Z Entering 'third_party/foxi'
2022-11-23T01:01:34.5626957Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:01:34.5661681Z Entering 'third_party/gloo'
2022-11-23T01:01:34.5697613Z Entering 'third_party/googletest'
2022-11-23T01:01:34.5731985Z Entering 'third_party/ideep'
2022-11-23T01:01:34.5770421Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:34.5806012Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:34.5846472Z Entering 'third_party/ios-cmake'
2022-11-23T01:01:34.5881736Z Entering 'third_party/ittapi'
2022-11-23T01:01:34.5916114Z Entering 'third_party/kineto'
2022-11-23T01:01:34.5952229Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:34.5985844Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:34.6020404Z Entering 'third_party/nccl/nccl'
2022-11-23T01:01:34.6055803Z Entering 'third_party/neon2sse'
2022-11-23T01:01:34.6092578Z Entering 'third_party/nlohmann'
2022-11-23T01:01:34.6129683Z Entering 'third_party/onnx'
2022-11-23T01:01:34.6179277Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:34.6216367Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:34.6254455Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:01:34.6291197Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:34.6330252Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:34.6365687Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:34.6399992Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:34.6439803Z Entering 'third_party/pocketfft'
2022-11-23T01:01:34.6475673Z Entering 'third_party/protobuf'
2022-11-23T01:01:34.6512543Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:34.6547906Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:34.6585681Z Entering 'third_party/psimd'
2022-11-23T01:01:34.6622098Z Entering 'third_party/pthreadpool'
2022-11-23T01:01:34.6657175Z Entering 'third_party/pybind11'
2022-11-23T01:01:34.6692617Z Entering 'third_party/python-enum'
2022-11-23T01:01:34.6726612Z Entering 'third_party/python-peachpy'
2022-11-23T01:01:34.6761265Z Entering 'third_party/python-six'
2022-11-23T01:01:34.6795982Z Entering 'third_party/sleef'
2022-11-23T01:01:34.6831213Z Entering 'third_party/tbb'
2022-11-23T01:01:34.6869431Z Entering 'third_party/tensorpipe'
2022-11-23T01:01:34.6906346Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:34.6940561Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:34.6974567Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:34.7008703Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:34.7043212Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:34.7080422Z Entering 'third_party/zstd'
2022-11-23T01:01:34.7136003Z ##[endgroup]
2022-11-23T01:01:34.7136711Z ##[group]Persisting credentials for submodules
2022-11-23T01:01:34.7137688Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:01:34.7395811Z Entering 'android/libs/fbjni'
2022-11-23T01:01:34.7429396Z Entering 'third_party/FP16'
2022-11-23T01:01:34.7465762Z Entering 'third_party/FXdiv'
2022-11-23T01:01:34.7501011Z Entering 'third_party/NNPACK'
2022-11-23T01:01:34.7535884Z Entering 'third_party/QNNPACK'
2022-11-23T01:01:34.7571117Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:01:34.7606733Z Entering 'third_party/XNNPACK'
2022-11-23T01:01:34.7652538Z Entering 'third_party/benchmark'
2022-11-23T01:01:34.7689615Z Entering 'third_party/cpuinfo'
2022-11-23T01:01:34.7725195Z Entering 'third_party/cub'
2022-11-23T01:01:34.7761879Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:01:34.7803158Z Entering 'third_party/cutlass'
2022-11-23T01:01:34.7845044Z Entering 'third_party/eigen'
2022-11-23T01:01:34.7882811Z Entering 'third_party/fbgemm'
2022-11-23T01:01:34.7918356Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:34.7953416Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:34.7990650Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:34.8027088Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:34.8062998Z Entering 'third_party/flatbuffers'
2022-11-23T01:01:34.8102476Z Entering 'third_party/fmt'
2022-11-23T01:01:34.8138705Z Entering 'third_party/foxi'
2022-11-23T01:01:34.8176459Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:01:34.8210948Z Entering 'third_party/gloo'
2022-11-23T01:01:34.8245818Z Entering 'third_party/googletest'
2022-11-23T01:01:34.8280414Z Entering 'third_party/ideep'
2022-11-23T01:01:34.8315966Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:34.8353180Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:34.8394462Z Entering 'third_party/ios-cmake'
2022-11-23T01:01:34.8430856Z Entering 'third_party/ittapi'
2022-11-23T01:01:34.8465163Z Entering 'third_party/kineto'
2022-11-23T01:01:34.8499602Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:34.8533675Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:34.8569868Z Entering 'third_party/nccl/nccl'
2022-11-23T01:01:34.8603727Z Entering 'third_party/neon2sse'
2022-11-23T01:01:34.8639170Z Entering 'third_party/nlohmann'
2022-11-23T01:01:34.8674021Z Entering 'third_party/onnx'
2022-11-23T01:01:34.8718988Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:34.8752694Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:34.8789461Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:01:34.8822803Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:34.8860930Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:34.8894692Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:34.8928001Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:34.8967607Z Entering 'third_party/pocketfft'
2022-11-23T01:01:34.9001901Z Entering 'third_party/protobuf'
2022-11-23T01:01:34.9038536Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:34.9072390Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:34.9108240Z Entering 'third_party/psimd'
2022-11-23T01:01:34.9144113Z Entering 'third_party/pthreadpool'
2022-11-23T01:01:34.9182902Z Entering 'third_party/pybind11'
2022-11-23T01:01:34.9221088Z Entering 'third_party/python-enum'
2022-11-23T01:01:34.9254715Z Entering 'third_party/python-peachpy'
2022-11-23T01:01:34.9288555Z Entering 'third_party/python-six'
2022-11-23T01:01:34.9321578Z Entering 'third_party/sleef'
2022-11-23T01:01:34.9355192Z Entering 'third_party/tbb'
2022-11-23T01:01:34.9391584Z Entering 'third_party/tensorpipe'
2022-11-23T01:01:34.9425645Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:34.9458871Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:34.9491155Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:34.9524513Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:34.9557251Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:34.9593608Z Entering 'third_party/zstd'
2022-11-23T01:01:34.9638404Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:01:34.9913877Z Entering 'android/libs/fbjni'
2022-11-23T01:01:34.9945056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:01:34.9959212Z Entering 'third_party/FP16'
2022-11-23T01:01:34.9992407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:01:35.0006192Z Entering 'third_party/FXdiv'
2022-11-23T01:01:35.0039467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:01:35.0053354Z Entering 'third_party/NNPACK'
2022-11-23T01:01:35.0084837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:01:35.0099952Z Entering 'third_party/QNNPACK'
2022-11-23T01:01:35.0133285Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:01:35.0147609Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:01:35.0179481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:01:35.0193917Z Entering 'third_party/XNNPACK'
2022-11-23T01:01:35.0225395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:01:35.0250129Z Entering 'third_party/benchmark'
2022-11-23T01:01:35.0282028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:01:35.0297174Z Entering 'third_party/cpuinfo'
2022-11-23T01:01:35.0329005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:01:35.0343641Z Entering 'third_party/cub'
2022-11-23T01:01:35.0374268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:01:35.0388828Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:01:35.0420976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:01:35.0439315Z Entering 'third_party/cutlass'
2022-11-23T01:01:35.0470193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:01:35.0492211Z Entering 'third_party/eigen'
2022-11-23T01:01:35.0523992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:01:35.0541147Z Entering 'third_party/fbgemm'
2022-11-23T01:01:35.0572874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:01:35.0587554Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:35.0617441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:01:35.0631595Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:35.0664085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:01:35.0678663Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:35.0709449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:01:35.0723496Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:35.0755222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:01:35.0770060Z Entering 'third_party/flatbuffers'
2022-11-23T01:01:35.0801971Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:01:35.0819168Z Entering 'third_party/fmt'
2022-11-23T01:01:35.0850945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:01:35.0865580Z Entering 'third_party/foxi'
2022-11-23T01:01:35.0897719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:01:35.0911830Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:01:35.0944080Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:01:35.0958409Z Entering 'third_party/gloo'
2022-11-23T01:01:35.0989597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:01:35.1004277Z Entering 'third_party/googletest'
2022-11-23T01:01:35.1036605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:01:35.1051869Z Entering 'third_party/ideep'
2022-11-23T01:01:35.1082367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:01:35.1097348Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:35.1127279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:01:35.1143847Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:35.1175107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:01:35.1195244Z Entering 'third_party/ios-cmake'
2022-11-23T01:01:35.1226941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:01:35.1240870Z Entering 'third_party/ittapi'
2022-11-23T01:01:35.1273200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:01:35.1286737Z Entering 'third_party/kineto'
2022-11-23T01:01:35.1318684Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:01:35.1332120Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:35.1363009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:01:35.1377905Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:35.1410117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:01:35.1426325Z Entering 'third_party/nccl/nccl'
2022-11-23T01:01:35.1458447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:01:35.1472973Z Entering 'third_party/neon2sse'
2022-11-23T01:01:35.1504588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:01:35.1517903Z Entering 'third_party/nlohmann'
2022-11-23T01:01:35.1550177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:01:35.1565689Z Entering 'third_party/onnx'
2022-11-23T01:01:35.1597725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:01:35.1623373Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.1655598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:01:35.1672162Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.1704263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:01:35.1721074Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:01:35.1750943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:01:35.1764294Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:35.1794675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:01:35.1811928Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.1843913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:01:35.1858141Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.1889951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:01:35.1903368Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.1934359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:01:35.1953479Z Entering 'third_party/pocketfft'
2022-11-23T01:01:35.1984747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:01:35.1998944Z Entering 'third_party/protobuf'
2022-11-23T01:01:35.2030380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:01:35.2047361Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:35.2078374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:01:35.2092089Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:35.2123033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:01:35.2139884Z Entering 'third_party/psimd'
2022-11-23T01:01:35.2171559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:01:35.2186046Z Entering 'third_party/pthreadpool'
2022-11-23T01:01:35.2217256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:01:35.2231428Z Entering 'third_party/pybind11'
2022-11-23T01:01:35.2263553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:01:35.2278410Z Entering 'third_party/python-enum'
2022-11-23T01:01:35.2309778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:01:35.2323989Z Entering 'third_party/python-peachpy'
2022-11-23T01:01:35.2356404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:01:35.2370837Z Entering 'third_party/python-six'
2022-11-23T01:01:35.2404025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:01:35.2419654Z Entering 'third_party/sleef'
2022-11-23T01:01:35.2451457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:01:35.2468227Z Entering 'third_party/tbb'
2022-11-23T01:01:35.2500639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:01:35.2517644Z Entering 'third_party/tensorpipe'
2022-11-23T01:01:35.2552038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:01:35.2566131Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:35.2600271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:01:35.2614745Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:35.2646324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:01:35.2661090Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:35.2693231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:01:35.2708067Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:35.2740824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:01:35.2753639Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.2787155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:01:35.2803915Z Entering 'third_party/zstd'
2022-11-23T01:01:35.2836382Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:01:35.3915048Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:01:35.4186570Z Entering 'android/libs/fbjni'
2022-11-23T01:01:35.4220470Z Entering 'third_party/FP16'
2022-11-23T01:01:35.4258165Z Entering 'third_party/FXdiv'
2022-11-23T01:01:35.4291765Z Entering 'third_party/NNPACK'
2022-11-23T01:01:35.4329404Z Entering 'third_party/QNNPACK'
2022-11-23T01:01:35.4364451Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:01:35.4400295Z Entering 'third_party/XNNPACK'
2022-11-23T01:01:35.4445900Z Entering 'third_party/benchmark'
2022-11-23T01:01:35.4482436Z Entering 'third_party/cpuinfo'
2022-11-23T01:01:35.4518544Z Entering 'third_party/cub'
2022-11-23T01:01:35.4552027Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:01:35.4593234Z Entering 'third_party/cutlass'
2022-11-23T01:01:35.4632524Z Entering 'third_party/eigen'
2022-11-23T01:01:35.4669016Z Entering 'third_party/fbgemm'
2022-11-23T01:01:35.4703036Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:35.4738149Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:35.4773426Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:35.4809613Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:35.4844650Z Entering 'third_party/flatbuffers'
2022-11-23T01:01:35.4881081Z Entering 'third_party/fmt'
2022-11-23T01:01:35.4915904Z Entering 'third_party/foxi'
2022-11-23T01:01:35.4950911Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:01:35.4986611Z Entering 'third_party/gloo'
2022-11-23T01:01:35.5022077Z Entering 'third_party/googletest'
2022-11-23T01:01:35.5056138Z Entering 'third_party/ideep'
2022-11-23T01:01:35.5090013Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:35.5126278Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:35.5167285Z Entering 'third_party/ios-cmake'
2022-11-23T01:01:35.5204321Z Entering 'third_party/ittapi'
2022-11-23T01:01:35.5241236Z Entering 'third_party/kineto'
2022-11-23T01:01:35.5277632Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:35.5311984Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:35.5348423Z Entering 'third_party/nccl/nccl'
2022-11-23T01:01:35.5383932Z Entering 'third_party/neon2sse'
2022-11-23T01:01:35.5417665Z Entering 'third_party/nlohmann'
2022-11-23T01:01:35.5451916Z Entering 'third_party/onnx'
2022-11-23T01:01:35.5497849Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.5531441Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.5567791Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:01:35.5603482Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:35.5643160Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.5678560Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.5715732Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.5757065Z Entering 'third_party/pocketfft'
2022-11-23T01:01:35.5792320Z Entering 'third_party/protobuf'
2022-11-23T01:01:35.5829950Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:35.5863942Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:35.5899162Z Entering 'third_party/psimd'
2022-11-23T01:01:35.5934104Z Entering 'third_party/pthreadpool'
2022-11-23T01:01:35.5969895Z Entering 'third_party/pybind11'
2022-11-23T01:01:35.6005869Z Entering 'third_party/python-enum'
2022-11-23T01:01:35.6040922Z Entering 'third_party/python-peachpy'
2022-11-23T01:01:35.6074875Z Entering 'third_party/python-six'
2022-11-23T01:01:35.6108595Z Entering 'third_party/sleef'
2022-11-23T01:01:35.6143277Z Entering 'third_party/tbb'
2022-11-23T01:01:35.6179214Z Entering 'third_party/tensorpipe'
2022-11-23T01:01:35.6213490Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:35.6247714Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:35.6280922Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:35.6316145Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:35.6350246Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.6388279Z Entering 'third_party/zstd'
2022-11-23T01:01:35.6440882Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:01:35.6699753Z Entering 'android/libs/fbjni'
2022-11-23T01:01:35.6733871Z Entering 'third_party/FP16'
2022-11-23T01:01:35.6771193Z Entering 'third_party/FXdiv'
2022-11-23T01:01:35.6837406Z Entering 'third_party/NNPACK'
2022-11-23T01:01:35.6843072Z Entering 'third_party/QNNPACK'
2022-11-23T01:01:35.6878280Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:01:35.6913528Z Entering 'third_party/XNNPACK'
2022-11-23T01:01:35.6962150Z Entering 'third_party/benchmark'
2022-11-23T01:01:35.7050159Z Entering 'third_party/cpuinfo'
2022-11-23T01:01:35.7083677Z Entering 'third_party/cub'
2022-11-23T01:01:35.7117197Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:01:35.7156583Z Entering 'third_party/cutlass'
2022-11-23T01:01:35.7195769Z Entering 'third_party/eigen'
2022-11-23T01:01:35.7230953Z Entering 'third_party/fbgemm'
2022-11-23T01:01:35.7276084Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:01:35.7310230Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:01:35.7345134Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:01:35.7379513Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:01:35.7414828Z Entering 'third_party/flatbuffers'
2022-11-23T01:01:35.7455360Z Entering 'third_party/fmt'
2022-11-23T01:01:35.7493040Z Entering 'third_party/foxi'
2022-11-23T01:01:35.7528776Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:01:35.7566044Z Entering 'third_party/gloo'
2022-11-23T01:01:35.7602004Z Entering 'third_party/googletest'
2022-11-23T01:01:35.7637834Z Entering 'third_party/ideep'
2022-11-23T01:01:35.7672653Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:01:35.7713048Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:01:35.7753296Z Entering 'third_party/ios-cmake'
2022-11-23T01:01:35.7789880Z Entering 'third_party/ittapi'
2022-11-23T01:01:35.7824172Z Entering 'third_party/kineto'
2022-11-23T01:01:35.7859621Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:01:35.7895987Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:01:35.7932974Z Entering 'third_party/nccl/nccl'
2022-11-23T01:01:35.7966711Z Entering 'third_party/neon2sse'
2022-11-23T01:01:35.8002238Z Entering 'third_party/nlohmann'
2022-11-23T01:01:35.8037892Z Entering 'third_party/onnx'
2022-11-23T01:01:35.8084721Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.8118181Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.8154556Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:01:35.8189247Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:01:35.8227409Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:01:35.8262458Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:01:35.8299194Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.8339717Z Entering 'third_party/pocketfft'
2022-11-23T01:01:35.8375044Z Entering 'third_party/protobuf'
2022-11-23T01:01:35.8414651Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:01:35.8450992Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:01:35.8488085Z Entering 'third_party/psimd'
2022-11-23T01:01:35.8525804Z Entering 'third_party/pthreadpool'
2022-11-23T01:01:35.8561864Z Entering 'third_party/pybind11'
2022-11-23T01:01:35.8597822Z Entering 'third_party/python-enum'
2022-11-23T01:01:35.8634244Z Entering 'third_party/python-peachpy'
2022-11-23T01:01:35.8669894Z Entering 'third_party/python-six'
2022-11-23T01:01:35.8705223Z Entering 'third_party/sleef'
2022-11-23T01:01:35.8741073Z Entering 'third_party/tbb'
2022-11-23T01:01:35.8780450Z Entering 'third_party/tensorpipe'
2022-11-23T01:01:35.8816319Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:01:35.8850865Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:01:35.8887203Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:01:35.8921705Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:01:35.8956246Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:01:35.8993063Z Entering 'third_party/zstd'
2022-11-23T01:01:35.9036644Z ##[endgroup]
2022-11-23T01:01:35.9080598Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:01:35.9108430Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:01:35.9229087Z Prepare all required actions
2022-11-23T01:01:35.9252366Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:01:35.9252565Z env:
2022-11-23T01:01:35.9252740Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:35.9252912Z ##[endgroup]
2022-11-23T01:01:35.9311366Z ##[group]Run set -euo pipefail
2022-11-23T01:01:35.9311603Z set -euo pipefail
2022-11-23T01:01:35.9311812Z function get_ec2_metadata() {
2022-11-23T01:01:35.9312045Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:01:35.9312397Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:01:35.9312696Z   category=$1
2022-11-23T01:01:35.9312936Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:01:35.9313151Z }
2022-11-23T01:01:35.9313351Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:01:35.9313632Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:01:35.9313896Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:01:35.9314145Z echo "system info $(uname -a)"
2022-11-23T01:01:35.9325150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:01:35.9325359Z env:
2022-11-23T01:01:35.9325535Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:35.9325723Z ##[endgroup]
2022-11-23T01:01:35.9402372Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:01:35.9450982Z instance-id: i-038a48515f8572324
2022-11-23T01:01:35.9500514Z instance-type: c5.2xlarge
2022-11-23T01:01:35.9507373Z system info Linux ip-10-0-6-115.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:01:35.9521213Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:01:35.9521496Z if systemctl is-active --quiet docker; then
2022-11-23T01:01:35.9521727Z     echo "Docker daemon is running...";
2022-11-23T01:01:35.9521939Z else
2022-11-23T01:01:35.9522174Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:01:35.9522389Z fi
2022-11-23T01:01:35.9532903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:01:35.9533125Z env:
2022-11-23T01:01:35.9533283Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:35.9533468Z ##[endgroup]
2022-11-23T01:01:35.9573602Z Docker daemon is running...
2022-11-23T01:01:35.9628180Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:01:35.9628533Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:01:35.9628818Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:01:35.9629196Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:01:35.9629547Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:01:35.9639403Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:01:35.9639622Z env:
2022-11-23T01:01:35.9639801Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:35.9639982Z   AWS_RETRY_MODE: standard
2022-11-23T01:01:35.9640258Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:01:35.9640477Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:01:35.9640653Z ##[endgroup]
2022-11-23T01:01:36.7102131Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:01:36.7102584Z Configure a credential helper to remove this warning. See
2022-11-23T01:01:36.7103328Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:01:36.7103628Z Login Succeeded
2022-11-23T01:01:36.7107872Z 
2022-11-23T01:01:36.7135545Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:01:36.7135853Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:01:36.7136206Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:01:36.7147112Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:01:36.7147329Z env:
2022-11-23T01:01:36.7147505Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:36.7147691Z ##[endgroup]
2022-11-23T01:01:36.7218815Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:01:36.7219067Z with:
2022-11-23T01:01:36.7219384Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:01:36.7219704Z env:
2022-11-23T01:01:36.7219878Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:36.7220052Z ##[endgroup]
2022-11-23T01:01:36.7276147Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:01:36.7276420Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:01:36.7276681Z # ignore output since only exit code is used for conditional
2022-11-23T01:01:36.7276960Z # only pull docker image if it's not available locally
2022-11-23T01:01:36.7277239Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:01:36.7277540Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:01:36.7277742Z fi
2022-11-23T01:01:36.7287808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:01:36.7288024Z env:
2022-11-23T01:01:36.7288198Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:01:36.7288541Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:01:36.7288868Z ##[endgroup]
2022-11-23T01:01:37.0001943Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7
2022-11-23T01:01:37.0002686Z eaead16dc43b: Pulling fs layer
2022-11-23T01:01:37.0002966Z 96afc5db69dc: Pulling fs layer
2022-11-23T01:01:37.0003208Z 9cd06b79192a: Pulling fs layer
2022-11-23T01:01:37.0003522Z ed66d74b7c20: Pulling fs layer
2022-11-23T01:01:37.0003909Z 42439e1d77c4: Pulling fs layer
2022-11-23T01:01:37.0004251Z 80bf4ab5d594: Pulling fs layer
2022-11-23T01:01:37.0004606Z 879beb9f8e9b: Pulling fs layer
2022-11-23T01:01:37.0004965Z 3f623e97839a: Pulling fs layer
2022-11-23T01:01:37.0005213Z d845eb9c30a0: Pulling fs layer
2022-11-23T01:01:37.0005390Z 8654f94b3130: Pulling fs layer
2022-11-23T01:01:37.0005567Z ed66d74b7c20: Waiting
2022-11-23T01:01:37.0005761Z 0c488367249b: Pulling fs layer
2022-11-23T01:01:37.0005978Z 0c91414faf6f: Pulling fs layer
2022-11-23T01:01:37.0006166Z 16b14aea0152: Pulling fs layer
2022-11-23T01:01:37.0006347Z 3f623e97839a: Waiting
2022-11-23T01:01:37.0006524Z 42439e1d77c4: Waiting
2022-11-23T01:01:37.0006735Z a2871863f729: Pulling fs layer
2022-11-23T01:01:37.0006914Z 80bf4ab5d594: Waiting
2022-11-23T01:01:37.0007074Z 879beb9f8e9b: Waiting
2022-11-23T01:01:37.0007252Z d703d3396c78: Pulling fs layer
2022-11-23T01:01:37.0007490Z acf65ded4ba0: Pulling fs layer
2022-11-23T01:01:37.0007666Z d845eb9c30a0: Waiting
2022-11-23T01:01:37.0007848Z 19d7da72e211: Pulling fs layer
2022-11-23T01:01:37.0008026Z 0c91414faf6f: Waiting
2022-11-23T01:01:37.0008180Z 8654f94b3130: Waiting
2022-11-23T01:01:37.0008361Z cb4b60b16a23: Pulling fs layer
2022-11-23T01:01:37.0008541Z 0c488367249b: Waiting
2022-11-23T01:01:37.0008712Z 6325a464626e: Pulling fs layer
2022-11-23T01:01:37.0008892Z b17aded2bf76: Pulling fs layer
2022-11-23T01:01:37.0009082Z 79f0454f85ca: Pulling fs layer
2022-11-23T01:01:37.0009275Z 5df784aed3a4: Pulling fs layer
2022-11-23T01:01:37.0009454Z 196c72536fc8: Pulling fs layer
2022-11-23T01:01:37.0009639Z e7cf72049767: Pulling fs layer
2022-11-23T01:01:37.0009815Z 16b14aea0152: Waiting
2022-11-23T01:01:37.0009982Z 7c4667efa9a5: Pulling fs layer
2022-11-23T01:01:37.0010172Z c62a223ffc22: Pulling fs layer
2022-11-23T01:01:37.0010349Z a2871863f729: Waiting
2022-11-23T01:01:37.0010516Z 824caeed1492: Pulling fs layer
2022-11-23T01:01:37.0010705Z 1762e4b0a4dc: Pulling fs layer
2022-11-23T01:01:37.0010886Z d703d3396c78: Waiting
2022-11-23T01:01:37.0011056Z 093e8e8bcad0: Pulling fs layer
2022-11-23T01:01:37.0011240Z cb4b60b16a23: Waiting
2022-11-23T01:01:37.0011422Z 133752b42076: Pulling fs layer
2022-11-23T01:01:37.0011808Z acf65ded4ba0: Waiting
2022-11-23T01:01:37.0012048Z 6325a464626e: Waiting
2022-11-23T01:01:37.0012215Z 79f0454f85ca: Waiting
2022-11-23T01:01:37.0012382Z 516276b2131d: Pulling fs layer
2022-11-23T01:01:37.0012564Z b17aded2bf76: Waiting
2022-11-23T01:01:37.0012750Z 3507de6a25b1: Pulling fs layer
2022-11-23T01:01:37.0012918Z 19d7da72e211: Waiting
2022-11-23T01:01:37.0013102Z 49880d42ecce: Pulling fs layer
2022-11-23T01:01:37.0013282Z e7cf72049767: Waiting
2022-11-23T01:01:37.0013442Z 7c4667efa9a5: Waiting
2022-11-23T01:01:37.0013613Z 824caeed1492: Waiting
2022-11-23T01:01:37.0013786Z 5df784aed3a4: Waiting
2022-11-23T01:01:37.0013940Z 196c72536fc8: Waiting
2022-11-23T01:01:37.0014126Z 4694407ca3de: Pulling fs layer
2022-11-23T01:01:37.0014310Z 1762e4b0a4dc: Waiting
2022-11-23T01:01:37.0014480Z b2c751e637f5: Pulling fs layer
2022-11-23T01:01:37.0014663Z 093e8e8bcad0: Waiting
2022-11-23T01:01:37.0014832Z 3507de6a25b1: Waiting
2022-11-23T01:01:37.0015149Z 133752b42076: Waiting
2022-11-23T01:01:37.0015320Z 516276b2131d: Waiting
2022-11-23T01:01:37.0015502Z 6114b441ddbf: Pulling fs layer
2022-11-23T01:01:37.0015671Z b2c751e637f5: Waiting
2022-11-23T01:01:37.0015851Z 52944c8173bb: Pulling fs layer
2022-11-23T01:01:37.0016042Z 1c1aab3b72ce: Pulling fs layer
2022-11-23T01:01:37.0016212Z 6114b441ddbf: Waiting
2022-11-23T01:01:37.0016399Z 9af1bdb6cccf: Pulling fs layer
2022-11-23T01:01:37.0016596Z 2b8671dbd244: Pulling fs layer
2022-11-23T01:01:37.0016811Z 0ccc0a3f9f55: Pulling fs layer
2022-11-23T01:01:37.0016991Z 6c09ee684801: Pulling fs layer
2022-11-23T01:01:37.0017185Z dde59f92df16: Pulling fs layer
2022-11-23T01:01:37.0017363Z 52944c8173bb: Waiting
2022-11-23T01:01:37.0017526Z 7108366bb357: Pulling fs layer
2022-11-23T01:01:37.0017712Z a17d3f0d245f: Pulling fs layer
2022-11-23T01:01:37.0017904Z af1f57b03dc1: Pulling fs layer
2022-11-23T01:01:37.0018078Z 0ccc0a3f9f55: Waiting
2022-11-23T01:01:37.0018253Z 1c1aab3b72ce: Waiting
2022-11-23T01:01:37.0018448Z 2b8671dbd244: Waiting
2022-11-23T01:01:37.0018656Z fedc8844da0c: Pulling fs layer
2022-11-23T01:01:37.0018838Z 6c09ee684801: Waiting
2022-11-23T01:01:37.0019047Z 4d9ee4a84202: Pulling fs layer
2022-11-23T01:01:37.0019216Z dde59f92df16: Waiting
2022-11-23T01:01:37.0019395Z 422bb402971f: Pulling fs layer
2022-11-23T01:01:37.0019575Z a17d3f0d245f: Waiting
2022-11-23T01:01:37.0019745Z 30d9fd19fcf8: Pulling fs layer
2022-11-23T01:01:37.0019924Z 4d9ee4a84202: Waiting
2022-11-23T01:01:37.0020089Z 7108366bb357: Waiting
2022-11-23T01:01:37.0020264Z e6ec9ff72352: Pulling fs layer
2022-11-23T01:01:37.0020453Z 422bb402971f: Waiting
2022-11-23T01:01:37.0020632Z b2dabb47a4a1: Pulling fs layer
2022-11-23T01:01:37.0020818Z 30d9fd19fcf8: Waiting
2022-11-23T01:01:37.0020986Z 78592748cf6a: Pulling fs layer
2022-11-23T01:01:37.0021169Z 2f62c284cd73: Pulling fs layer
2022-11-23T01:01:37.0021349Z e6ec9ff72352: Waiting
2022-11-23T01:01:37.0021517Z d25a76b3a031: Pulling fs layer
2022-11-23T01:01:37.0021706Z 5a50d7c62a26: Pulling fs layer
2022-11-23T01:01:37.0021900Z cd6d94bb12b1: Pulling fs layer
2022-11-23T01:01:37.0022086Z 2fba50966c3e: Pulling fs layer
2022-11-23T01:01:37.0022272Z b2dabb47a4a1: Waiting
2022-11-23T01:01:37.0022441Z 2f62c284cd73: Waiting
2022-11-23T01:01:37.0022595Z 78592748cf6a: Waiting
2022-11-23T01:01:37.0022761Z cd6d94bb12b1: Waiting
2022-11-23T01:01:37.0022930Z d25a76b3a031: Waiting
2022-11-23T01:01:37.0023082Z 5a50d7c62a26: Waiting
2022-11-23T01:01:37.0665500Z 96afc5db69dc: Verifying Checksum
2022-11-23T01:01:37.0665967Z 96afc5db69dc: Download complete
2022-11-23T01:01:37.1472183Z ed66d74b7c20: Download complete
2022-11-23T01:01:37.2255054Z 42439e1d77c4: Verifying Checksum
2022-11-23T01:01:37.2255343Z 42439e1d77c4: Download complete
2022-11-23T01:01:37.2899982Z 80bf4ab5d594: Verifying Checksum
2022-11-23T01:01:37.2900373Z 80bf4ab5d594: Download complete
2022-11-23T01:01:37.3436151Z eaead16dc43b: Verifying Checksum
2022-11-23T01:01:37.3436766Z eaead16dc43b: Download complete
2022-11-23T01:01:37.3930402Z 879beb9f8e9b: Verifying Checksum
2022-11-23T01:01:37.3930864Z 879beb9f8e9b: Download complete
2022-11-23T01:01:37.4135121Z 3f623e97839a: Download complete
2022-11-23T01:01:37.4543777Z d845eb9c30a0: Download complete
2022-11-23T01:01:37.4723741Z 8654f94b3130: Download complete
2022-11-23T01:01:37.5525576Z 0c91414faf6f: Download complete
2022-11-23T01:01:37.6495399Z 16b14aea0152: Verifying Checksum
2022-11-23T01:01:37.6495758Z 16b14aea0152: Download complete
2022-11-23T01:01:38.0611011Z eaead16dc43b: Pull complete
2022-11-23T01:01:38.1666018Z 96afc5db69dc: Pull complete
2022-11-23T01:01:38.3241874Z 0c488367249b: Verifying Checksum
2022-11-23T01:01:38.3242273Z 0c488367249b: Download complete
2022-11-23T01:01:38.4043256Z d703d3396c78: Download complete
2022-11-23T01:01:38.4811466Z acf65ded4ba0: Verifying Checksum
2022-11-23T01:01:38.4811875Z acf65ded4ba0: Download complete
2022-11-23T01:01:38.9930431Z 19d7da72e211: Verifying Checksum
2022-11-23T01:01:38.9930805Z 19d7da72e211: Download complete
2022-11-23T01:01:39.1504633Z 6325a464626e: Verifying Checksum
2022-11-23T01:01:39.1504913Z 6325a464626e: Download complete
2022-11-23T01:01:39.2357651Z b17aded2bf76: Download complete
2022-11-23T01:01:39.3136122Z 79f0454f85ca: Verifying Checksum
2022-11-23T01:01:39.3136438Z 79f0454f85ca: Download complete
2022-11-23T01:01:39.3905835Z 5df784aed3a4: Download complete
2022-11-23T01:01:39.4683909Z 196c72536fc8: Verifying Checksum
2022-11-23T01:01:39.4684150Z 196c72536fc8: Download complete
2022-11-23T01:01:39.5444713Z e7cf72049767: Download complete
2022-11-23T01:01:40.1886098Z 9cd06b79192a: Download complete
2022-11-23T01:01:40.2778374Z c62a223ffc22: Download complete
2022-11-23T01:01:40.3648662Z 824caeed1492: Verifying Checksum
2022-11-23T01:01:40.3649066Z 824caeed1492: Download complete
2022-11-23T01:01:40.4736445Z 1762e4b0a4dc: Verifying Checksum
2022-11-23T01:01:40.4736723Z 1762e4b0a4dc: Download complete
2022-11-23T01:01:40.4820149Z 7c4667efa9a5: Verifying Checksum
2022-11-23T01:01:40.4820664Z 7c4667efa9a5: Download complete
2022-11-23T01:01:40.5373238Z 093e8e8bcad0: Verifying Checksum
2022-11-23T01:01:40.5373589Z 093e8e8bcad0: Download complete
2022-11-23T01:01:40.5585933Z 133752b42076: Verifying Checksum
2022-11-23T01:01:40.5586462Z 133752b42076: Download complete
2022-11-23T01:01:40.6259857Z 3507de6a25b1: Download complete
2022-11-23T01:01:40.6996395Z 49880d42ecce: Download complete
2022-11-23T01:01:40.7743885Z 4694407ca3de: Verifying Checksum
2022-11-23T01:01:40.7744318Z 4694407ca3de: Download complete
2022-11-23T01:01:40.8440634Z b2c751e637f5: Verifying Checksum
2022-11-23T01:01:40.8441041Z b2c751e637f5: Download complete
2022-11-23T01:01:40.9248178Z 6114b441ddbf: Verifying Checksum
2022-11-23T01:01:40.9248950Z 6114b441ddbf: Download complete
2022-11-23T01:01:40.9745076Z 52944c8173bb: Verifying Checksum
2022-11-23T01:01:40.9745495Z 52944c8173bb: Download complete
2022-11-23T01:01:41.0544285Z 1c1aab3b72ce: Verifying Checksum
2022-11-23T01:01:41.0544691Z 1c1aab3b72ce: Download complete
2022-11-23T01:01:41.1354027Z 9af1bdb6cccf: Verifying Checksum
2022-11-23T01:01:41.1354475Z 9af1bdb6cccf: Download complete
2022-11-23T01:01:41.2440269Z 2b8671dbd244: Verifying Checksum
2022-11-23T01:01:41.2440702Z 2b8671dbd244: Download complete
2022-11-23T01:01:41.3121201Z 0ccc0a3f9f55: Download complete
2022-11-23T01:01:41.3943508Z 6c09ee684801: Verifying Checksum
2022-11-23T01:01:41.3945145Z 6c09ee684801: Download complete
2022-11-23T01:01:41.4719318Z dde59f92df16: Download complete
2022-11-23T01:01:41.9412837Z 7108366bb357: Verifying Checksum
2022-11-23T01:01:41.9413306Z 7108366bb357: Download complete
2022-11-23T01:01:42.0129858Z a17d3f0d245f: Download complete
2022-11-23T01:01:42.1077860Z af1f57b03dc1: Verifying Checksum
2022-11-23T01:01:42.1078332Z af1f57b03dc1: Download complete
2022-11-23T01:01:42.2019581Z fedc8844da0c: Verifying Checksum
2022-11-23T01:01:42.2019854Z fedc8844da0c: Download complete
2022-11-23T01:01:42.2784220Z 4d9ee4a84202: Verifying Checksum
2022-11-23T01:01:42.2784591Z 4d9ee4a84202: Download complete
2022-11-23T01:01:42.5282459Z 422bb402971f: Verifying Checksum
2022-11-23T01:01:42.5283402Z 422bb402971f: Download complete
2022-11-23T01:01:42.5940854Z 30d9fd19fcf8: Verifying Checksum
2022-11-23T01:01:42.5941769Z 30d9fd19fcf8: Download complete
2022-11-23T01:01:42.6520686Z e6ec9ff72352: Download complete
2022-11-23T01:01:42.7838375Z 516276b2131d: Download complete
2022-11-23T01:01:42.8527027Z 78592748cf6a: Verifying Checksum
2022-11-23T01:01:42.8527410Z 78592748cf6a: Download complete
2022-11-23T01:01:42.9346328Z 2f62c284cd73: Download complete
2022-11-23T01:01:43.0123510Z d25a76b3a031: Download complete
2022-11-23T01:01:43.1358520Z 5a50d7c62a26: Download complete
2022-11-23T01:01:43.1997385Z cd6d94bb12b1: Verifying Checksum
2022-11-23T01:01:43.1997810Z cd6d94bb12b1: Download complete
2022-11-23T01:01:43.7964349Z 2fba50966c3e: Verifying Checksum
2022-11-23T01:01:43.7964785Z 2fba50966c3e: Download complete
2022-11-23T01:01:46.9547565Z 9cd06b79192a: Pull complete
2022-11-23T01:01:47.1240019Z ed66d74b7c20: Pull complete
2022-11-23T01:01:47.2994677Z b2dabb47a4a1: Verifying Checksum
2022-11-23T01:01:47.2994955Z b2dabb47a4a1: Download complete
2022-11-23T01:01:47.3144591Z 42439e1d77c4: Pull complete
2022-11-23T01:01:47.4643446Z 80bf4ab5d594: Pull complete
2022-11-23T01:01:47.6933302Z 879beb9f8e9b: Pull complete
2022-11-23T01:01:47.9193471Z 3f623e97839a: Pull complete
2022-11-23T01:01:48.1518169Z d845eb9c30a0: Pull complete
2022-11-23T01:01:48.3999682Z 8654f94b3130: Pull complete
2022-11-23T01:01:50.2584003Z a2871863f729: Verifying Checksum
2022-11-23T01:01:50.2584433Z a2871863f729: Download complete
2022-11-23T01:01:51.2945432Z 0c488367249b: Pull complete
2022-11-23T01:01:51.7749057Z 0c91414faf6f: Pull complete
2022-11-23T01:01:51.9539892Z 16b14aea0152: Pull complete
2022-11-23T01:02:13.8215750Z a2871863f729: Pull complete
2022-11-23T01:02:14.2043681Z d703d3396c78: Pull complete
2022-11-23T01:02:14.5501183Z acf65ded4ba0: Pull complete
2022-11-23T01:02:15.3372003Z 19d7da72e211: Pull complete
2022-11-23T01:02:15.4972781Z cb4b60b16a23: Pull complete
2022-11-23T01:02:15.6680147Z 6325a464626e: Pull complete
2022-11-23T01:02:15.8000230Z b17aded2bf76: Pull complete
2022-11-23T01:02:15.9749733Z 79f0454f85ca: Pull complete
2022-11-23T01:02:16.1062961Z 5df784aed3a4: Pull complete
2022-11-23T01:02:16.3197701Z 196c72536fc8: Pull complete
2022-11-23T01:02:16.5639771Z e7cf72049767: Pull complete
2022-11-23T01:02:18.5050463Z 7c4667efa9a5: Pull complete
2022-11-23T01:02:18.7236680Z c62a223ffc22: Pull complete
2022-11-23T01:02:18.9520207Z 824caeed1492: Pull complete
2022-11-23T01:02:19.1409604Z 1762e4b0a4dc: Pull complete
2022-11-23T01:02:19.2381456Z 093e8e8bcad0: Pull complete
2022-11-23T01:02:19.3502395Z 133752b42076: Pull complete
2022-11-23T01:02:23.8936425Z 516276b2131d: Pull complete
2022-11-23T01:02:24.0382281Z 3507de6a25b1: Pull complete
2022-11-23T01:02:24.1425294Z 49880d42ecce: Pull complete
2022-11-23T01:02:24.2427086Z 4694407ca3de: Pull complete
2022-11-23T01:02:24.3227540Z b2c751e637f5: Pull complete
2022-11-23T01:02:24.4176018Z 6114b441ddbf: Pull complete
2022-11-23T01:02:24.5140707Z 52944c8173bb: Pull complete
2022-11-23T01:02:24.6024584Z 1c1aab3b72ce: Pull complete
2022-11-23T01:02:24.6944052Z 9af1bdb6cccf: Pull complete
2022-11-23T01:02:24.7863968Z 2b8671dbd244: Pull complete
2022-11-23T01:02:24.8653608Z 0ccc0a3f9f55: Pull complete
2022-11-23T01:02:24.9739767Z 6c09ee684801: Pull complete
2022-11-23T01:02:25.0774825Z dde59f92df16: Pull complete
2022-11-23T01:02:26.3252285Z 7108366bb357: Pull complete
2022-11-23T01:02:26.5163919Z a17d3f0d245f: Pull complete
2022-11-23T01:02:26.7078456Z af1f57b03dc1: Pull complete
2022-11-23T01:02:26.9399665Z fedc8844da0c: Pull complete
2022-11-23T01:02:27.1727533Z 4d9ee4a84202: Pull complete
2022-11-23T01:02:27.6266491Z 422bb402971f: Pull complete
2022-11-23T01:02:27.8453335Z 30d9fd19fcf8: Pull complete
2022-11-23T01:02:28.0884738Z e6ec9ff72352: Pull complete
2022-11-23T01:02:32.6946923Z b2dabb47a4a1: Pull complete
2022-11-23T01:02:32.8092589Z 78592748cf6a: Pull complete
2022-11-23T01:02:32.9207892Z 2f62c284cd73: Pull complete
2022-11-23T01:02:33.0279948Z d25a76b3a031: Pull complete
2022-11-23T01:02:33.1120175Z 5a50d7c62a26: Pull complete
2022-11-23T01:02:33.2233260Z cd6d94bb12b1: Pull complete
2022-11-23T01:02:34.6926917Z 2fba50966c3e: Pull complete
2022-11-23T01:02:34.7055745Z Digest: sha256:128a8060bc8d0d2006ae18b574265c3aa935b731c43947b53188b825fa718d6b
2022-11-23T01:02:34.7091149Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:02:34.7115879Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:02:34.7171399Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:02:34.7171763Z python3 -m pip install psutil==5.9.1
2022-11-23T01:02:34.7171989Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:02:34.7172247Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:02:34.7172526Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:02:34.7183537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:02:34.7183754Z env:
2022-11-23T01:02:34.7183929Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:34.7184103Z ##[endgroup]
2022-11-23T01:02:34.9279774Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:02:34.9462589Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-11-23T01:02:35.3533128Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:02:35.3713803Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-11-23T01:02:35.5772637Z Prepare all required actions
2022-11-23T01:02:35.5772906Z Getting action download info
2022-11-23T01:02:35.7746677Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:02:36.0472399Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:02:36.1677176Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:02:36.1677402Z with:
2022-11-23T01:02:36.1677590Z   name: linux-focal-py3.7-gcc7
2022-11-23T01:02:36.1677763Z env:
2022-11-23T01:02:36.1677933Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:36.1678119Z ##[endgroup]
2022-11-23T01:02:36.1701713Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:02:36.1701918Z with:
2022-11-23T01:02:36.1702106Z   name: linux-focal-py3.7-gcc7
2022-11-23T01:02:36.1702316Z   s3-bucket: gha-artifacts
2022-11-23T01:02:36.1702493Z   region: us-east-1
2022-11-23T01:02:36.1702659Z env:
2022-11-23T01:02:36.1702858Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:36.1703050Z ##[endgroup]
2022-11-23T01:02:36.5951915Z Found 1 objects with prefix pytorch/pytorch/3528293562/linux-focal-py3.7-gcc7/
2022-11-23T01:02:36.5952391Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:02:39.6934444Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:02:39.6934743Z 
2022-11-23T01:02:39.6949518Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:02:39.6957489Z Artifact download has finished successfully
2022-11-23T01:02:39.7057049Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:02:39.7057278Z unzip -o artifacts.zip
2022-11-23T01:02:39.7068361Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:02:39.7068569Z env:
2022-11-23T01:02:39.7068807Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:39.7068997Z ##[endgroup]
2022-11-23T01:02:39.7104618Z Archive:  artifacts.zip
2022-11-23T01:02:39.7105686Z    creating: dist/
2022-11-23T01:02:40.4660014Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl  
2022-11-23T01:02:40.4660729Z    creating: build/custom_test_artifacts/
2022-11-23T01:02:40.4661052Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:02:40.4661413Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:02:40.4663465Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:02:40.4664194Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:02:40.4664976Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:02:40.4665600Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:02:40.4666005Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:02:40.4667725Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:02:40.4669517Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:02:40.4670304Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:02:40.4670748Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:02:40.4672555Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:02:40.4674264Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:02:40.4676169Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:02:40.4676996Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:02:40.4678327Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:02:40.4679223Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:02:40.4680007Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:02:40.4680745Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:02:40.4681505Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:02:40.4682273Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:02:40.4683103Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:02:40.4683977Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:02:40.4684662Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:02:40.4685460Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:02:40.4686291Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:02:40.4686828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:02:40.4687281Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:02:40.4687720Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:02:40.4688163Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:02:40.4704970Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:02:40.4800647Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:02:40.4801416Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:02:40.4802382Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:02:40.4803248Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:02:40.4804098Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:02:40.4804934Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:02:40.4805504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:02:40.4805949Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:02:40.4806408Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:02:40.4806867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:02:40.4807322Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:02:40.4820841Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:02:40.4884061Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:02:40.4884955Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:02:40.4885964Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:02:40.4886737Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:02:40.4887355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:02:40.4887793Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:02:40.4888521Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:02:40.4889187Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:02:40.4889763Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:02:40.4961183Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:02:40.5009834Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:02:40.5010470Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:02:40.5010892Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:02:40.5012938Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:02:40.5013644Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:02:40.5014389Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:02:40.5014809Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:02:40.5015372Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:02:40.5016230Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:02:40.5017477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:02:40.5018252Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:02:40.5018687Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:02:40.5019942Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:02:40.5021219Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:02:40.5022590Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:02:40.5023432Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:02:40.5024516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:02:40.5025358Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:02:40.5026086Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:02:40.5026820Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:02:40.5027544Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:02:40.5028293Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:02:40.5029050Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:02:40.5029908Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:02:40.5030727Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:02:40.5031455Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:02:40.5031904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:02:40.5032425Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:02:40.5032863Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:02:40.5033310Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:02:40.5033753Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:02:40.5046661Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:02:40.5095450Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:02:40.5096314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:02:40.5097158Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:02:40.5097931Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:02:40.5098557Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:02:40.5098995Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:02:40.5099699Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:02:40.5100376Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:02:40.5100962Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:02:40.5138448Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:02:40.5139120Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:02:40.5139528Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:02:40.5141503Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:02:40.5142284Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:02:40.5142932Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:02:40.5143354Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:02:40.5143875Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:02:40.5144714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:02:40.5145705Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:02:40.5146513Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:02:40.5146961Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:02:40.5148172Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:02:40.5149402Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:02:40.5150775Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:02:40.5151621Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:02:40.5152674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:02:40.5153577Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:02:40.5154353Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:02:40.5155203Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:02:40.5155987Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:02:40.5156788Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:02:40.5157651Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:02:40.5158551Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:02:40.5159293Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:02:40.5159757Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:02:40.5160218Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:02:40.5160691Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:02:40.5161169Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:02:40.5161644Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:02:40.5162478Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:02:40.5163024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:02:40.5275877Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:02:40.5276743Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:02:40.5277635Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:02:40.5278579Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:02:40.5279478Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:02:40.5280417Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:02:40.5280894Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:02:40.5281385Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:02:40.5281875Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:02:40.5282363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:02:40.5282829Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:02:40.5295962Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:02:40.5340865Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:02:40.5341807Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:02:40.5342722Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:02:40.5343518Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:02:40.5344046Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:02:40.5344911Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:02:40.5345669Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:02:40.5346276Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:02:40.5346677Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:02:40.5437495Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:02:40.5473090Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:02:40.5473586Z    creating: build/lib/
2022-11-23T01:02:40.5473924Z   inflating: build/lib/libclog.a     
2022-11-23T01:02:40.5525205Z   inflating: build/lib/libgtest.a    
2022-11-23T01:02:40.5533160Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:02:40.5604742Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:02:40.5611807Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:02:40.5694598Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:02:40.5719495Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:02:40.5809893Z   inflating: build/lib/libgloo.a     
2022-11-23T01:02:40.5870278Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:02:40.6284946Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:02:40.6309946Z   inflating: build/lib/libfmt.a      
2022-11-23T01:02:40.6310495Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:02:40.6372215Z   inflating: build/lib/libc10.so     
2022-11-23T01:02:40.6373320Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:02:40.6380807Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:02:40.6387855Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:02:40.6399996Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:02:40.6402364Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:02:40.6848259Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:02:40.6862461Z   inflating: build/lib/libgmock.a    
2022-11-23T01:02:40.6879945Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:02:40.6880280Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:02:40.6880956Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:02:41.4423206Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:02:41.4928903Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:02:41.4946265Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:02:41.6132591Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:02:41.7017288Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:02:41.7017608Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:02:41.7245904Z   inflating: build/lib/libkineto.a   
2022-11-23T01:02:41.7356088Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:02:41.7391002Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:02:41.7428036Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:02:41.7953099Z   inflating: build/lib/libonnx.a     
2022-11-23T01:02:43.6243119Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:02:43.6243781Z   inflating: build/lib/libtorch.so   
2022-11-23T01:02:43.6290759Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:02:43.6309472Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:02:43.6333675Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:02:43.6337299Z   inflating: build/lib/libshm.so     
2022-11-23T01:02:43.7658053Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:02:43.7688401Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:02:43.7688810Z    creating: build/bin/
2022-11-23T01:02:43.7689183Z    creating: build/bin/CMakeFiles/
2022-11-23T01:02:43.7689685Z   inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:02:43.7690166Z  extracting: build/bin/CMakeFiles/progress.marks  
2022-11-23T01:02:43.7732503Z   inflating: build/bin/weakref_test  
2022-11-23T01:02:43.7733224Z   inflating: build/bin/Makefile      
2022-11-23T01:02:43.7734981Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:02:43.7735438Z   inflating: build/bin/cmake_install.cmake  
2022-11-23T01:02:43.7793412Z   inflating: build/bin/vmap_test     
2022-11-23T01:02:43.7793849Z   inflating: build/bin/CTestTestfile.cmake  
2022-11-23T01:02:43.7835118Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:02:43.7878476Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:02:43.7920748Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:02:43.7970365Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:02:43.8010798Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:02:43.8052244Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:02:43.8099356Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:02:43.8146175Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:02:43.8186406Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:02:43.8234570Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:02:43.8279277Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:02:43.8322197Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:02:43.8362580Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:02:43.8403731Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:02:43.8445575Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:02:43.8493133Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:02:43.8546528Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:02:43.8667588Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:02:43.8709587Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:02:43.8757466Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:02:43.8801491Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:02:43.8844252Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:02:43.8884193Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:02:43.8927250Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:02:43.8974265Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:02:43.9019695Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:02:43.9066352Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:02:43.9110210Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:02:43.9201854Z   inflating: build/bin/c10_either_test  
2022-11-23T01:02:43.9243818Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:02:43.9386089Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:02:43.9428145Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:02:43.9476716Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:02:43.9539549Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:02:43.9591603Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:02:43.9637385Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:02:43.9686932Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:02:43.9734261Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:02:43.9777256Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:02:43.9824529Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:02:44.0235691Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:02:44.0646330Z   inflating: build/bin/protoc        
2022-11-23T01:02:44.0892011Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:02:44.1168886Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:02:44.1219779Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:02:44.1264702Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:02:44.1309703Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:02:44.1312261Z   inflating: build/bin/example_allreduce  
2022-11-23T01:02:44.1367328Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:02:44.1412118Z   inflating: build/bin/Dimname_test  
2022-11-23T01:02:44.1473349Z   inflating: build/bin/Dict_test     
2022-11-23T01:02:44.1530865Z   inflating: build/bin/static_runtime_bench  
2022-11-23T01:02:44.1584183Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:02:44.1632170Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:02:44.1681632Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:02:44.1897354Z   inflating: build/bin/static_runtime_test  
2022-11-23T01:02:44.1946718Z   inflating: build/bin/atest         
2022-11-23T01:02:44.1998333Z   inflating: build/bin/basic         
2022-11-23T01:02:44.2043460Z   inflating: build/bin/broadcast_test  
2022-11-23T01:02:44.2092397Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:02:44.2136357Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:02:44.2178160Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:02:44.2219713Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:02:44.2293802Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:02:44.2342736Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:02:44.2389608Z   inflating: build/bin/half_test     
2022-11-23T01:02:44.2430831Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:02:44.2510718Z   inflating: build/bin/ivalue_test   
2022-11-23T01:02:44.2555686Z   inflating: build/bin/memory_format_test  
2022-11-23T01:02:44.2600783Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:02:44.2645780Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:02:44.2688231Z   inflating: build/bin/operator_name_test  
2022-11-23T01:02:44.2732067Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:02:44.2779185Z   inflating: build/bin/native_test   
2022-11-23T01:02:44.2821123Z   inflating: build/bin/operators_test  
2022-11-23T01:02:44.2865159Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:02:44.2920012Z   inflating: build/bin/pow_test      
2022-11-23T01:02:44.2968029Z   inflating: build/bin/quantized_test  
2022-11-23T01:02:44.3010526Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:02:44.3051925Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:02:44.3099629Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:02:44.3147890Z   inflating: build/bin/scalar_test   
2022-11-23T01:02:44.3191767Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:02:44.3257812Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:02:44.3259928Z   inflating: build/bin/thread_init_test  
2022-11-23T01:02:44.3306747Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:02:44.3353725Z   inflating: build/bin/test_parallel  
2022-11-23T01:02:44.3395131Z   inflating: build/bin/variant_test  
2022-11-23T01:02:44.3445817Z   inflating: build/bin/type_test     
2022-11-23T01:02:44.3489512Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:02:44.3532823Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:02:44.3583228Z   inflating: build/bin/IListRef_test  
2022-11-23T01:02:44.3624123Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:02:44.3716085Z   inflating: build/bin/List_test     
2022-11-23T01:02:44.3818744Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:02:44.3899803Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:02:44.3954567Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:02:44.4062858Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:02:44.4149831Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:02:44.4200130Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:02:44.4281038Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:02:44.4321295Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:02:44.4363429Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:02:44.4407906Z   inflating: build/bin/inline_container_test  
2022-11-23T01:02:44.4651156Z   inflating: build/bin/op_registration_test  
2022-11-23T01:02:44.4698665Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:02:44.4712407Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:02:44.4757531Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:02:44.4816614Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:02:44.4818526Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:02:44.5502821Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:02:44.5560776Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:02:44.5569498Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:02:44.5869411Z   inflating: build/bin/test_lazy     
2022-11-23T01:02:44.5873956Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:02:44.6890807Z   inflating: build/bin/test_api      
2022-11-23T01:02:44.7387082Z   inflating: build/bin/test_jit      
2022-11-23T01:02:44.7388585Z   inflating: .pytorch-test-times.json  
2022-11-23T01:02:44.7408952Z ##[group]Run df -H
2022-11-23T01:02:44.7409143Z df -H
2022-11-23T01:02:44.7420290Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:02:44.7420499Z env:
2022-11-23T01:02:44.7420675Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:44.7420865Z ##[endgroup]
2022-11-23T01:02:44.7452144Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:02:44.7452571Z devtmpfs        8.2G     0  8.2G   0% /dev
2022-11-23T01:02:44.7452945Z tmpfs           8.2G   41k  8.2G   1% /dev/shm
2022-11-23T01:02:44.7453187Z tmpfs           8.2G  431k  8.2G   1% /run
2022-11-23T01:02:44.7453397Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2022-11-23T01:02:44.7453600Z /dev/nvme0n1p1  162G   16G  146G  10% /
2022-11-23T01:02:44.7476963Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:02:44.7477379Z .github/scripts/parse_ref.py
2022-11-23T01:02:44.7491760Z shell: /usr/bin/bash -e {0}
2022-11-23T01:02:44.7492058Z env:
2022-11-23T01:02:44.7492328Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:44.7492628Z ##[endgroup]
2022-11-23T01:02:44.7714333Z ##[group]Run set -x
2022-11-23T01:02:44.7714602Z set -x
2022-11-23T01:02:44.7714754Z 
2022-11-23T01:02:44.7714949Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:02:44.7715205Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:02:44.7715448Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:02:44.7715685Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:02:44.7715881Z else
2022-11-23T01:02:44.7716069Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:02:44.7716342Z fi
2022-11-23T01:02:44.7716498Z 
2022-11-23T01:02:44.7716720Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:02:44.7716954Z 
2022-11-23T01:02:44.7717165Z # sanitize the input commit message and PR body here:
2022-11-23T01:02:44.7717377Z #
2022-11-23T01:02:44.7717658Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:02:44.7718020Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:02:44.7718330Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:02:44.7718559Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:02:44.7718740Z 
2022-11-23T01:02:44.7718985Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:02:44.7719259Z # wreak havoc internally
2022-11-23T01:02:44.7719495Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:02:44.7719722Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:02:44.7719910Z 
2022-11-23T01:02:44.7720135Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:02:44.7720416Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:02:44.7720688Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:02:44.7720927Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:02:44.7721146Z container_name=$(docker run \
2022-11-23T01:02:44.7721332Z   ${GPU_FLAG:-} \
2022-11-23T01:02:44.7721526Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:02:44.7721721Z   -e PR_NUMBER \
2022-11-23T01:02:44.7721902Z   -e GITHUB_ACTIONS \
2022-11-23T01:02:44.7722095Z   -e BASE_SHA \
2022-11-23T01:02:44.7722272Z   -e BRANCH \
2022-11-23T01:02:44.7722437Z   -e SHA1 \
2022-11-23T01:02:44.7722624Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:02:44.7722819Z   -e IN_WHEEL_TEST \
2022-11-23T01:02:44.7722994Z   -e SHARD_NUMBER \
2022-11-23T01:02:44.7723185Z   -e TEST_CONFIG \
2022-11-23T01:02:44.7723373Z   -e NUM_TEST_SHARDS \
2022-11-23T01:02:44.7723558Z   -e PR_BODY \
2022-11-23T01:02:44.7723734Z   -e COMMIT_MESSAGES \
2022-11-23T01:02:44.7723945Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:02:44.7724171Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:02:44.7724365Z   -e PR_LABELS \
2022-11-23T01:02:44.7724576Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:02:44.7724782Z   -e SCCACHE_BUCKET \
2022-11-23T01:02:44.7724971Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:02:44.7725167Z   -e XLA_CUDA \
2022-11-23T01:02:44.7725372Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:02:44.7725594Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:02:44.7725832Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:02:44.7726085Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:02:44.7726318Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:02:44.7726530Z   --security-opt seccomp=unconfined \
2022-11-23T01:02:44.7726805Z   --cap-add=SYS_PTRACE \
2022-11-23T01:02:44.7727092Z   --ipc=host \
2022-11-23T01:02:44.7727272Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:02:44.7727458Z   --tty \
2022-11-23T01:02:44.7727630Z   --detach \
2022-11-23T01:02:44.7727813Z   --name="${container_name}" \
2022-11-23T01:02:44.7728010Z   --user jenkins \
2022-11-23T01:02:44.7728241Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:02:44.7728478Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:02:44.7728723Z   "${DOCKER_IMAGE}"
2022-11-23T01:02:44.7728897Z )
2022-11-23T01:02:44.7729119Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:02:44.7729438Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:02:44.7739909Z shell: /usr/bin/bash -e {0}
2022-11-23T01:02:44.7740090Z env:
2022-11-23T01:02:44.7740257Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:02:44.7740490Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-11-23T01:02:44.7740702Z   PR_NUMBER: 
2022-11-23T01:02:44.7740860Z   BRANCH: master
2022-11-23T01:02:44.7741062Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:02:44.7741293Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:02:44.7741496Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:02:44.7741703Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:02:44.7741907Z   TEST_CONFIG: distributed
2022-11-23T01:02:44.7742074Z   SHARD_NUMBER: 1
2022-11-23T01:02:44.7742250Z   NUM_TEST_SHARDS: 2
2022-11-23T01:02:44.7742420Z   PR_BODY: 
2022-11-23T01:02:44.7742631Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:02:44.7742874Z   SCCACHE_S3_KEY_PREFIX: pull
2022-11-23T01:02:44.7743056Z   SHM_SIZE: 1g
2022-11-23T01:02:44.7743392Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:02:44.7743699Z   XLA_CUDA: 
2022-11-23T01:02:44.7743956Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:02:44.7744234Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:02:44.7744437Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2022-11-23T01:02:44.7744635Z ##[endgroup]
2022-11-23T01:02:44.7767757Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:02:44.7768446Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]]
2022-11-23T01:02:44.7768759Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:02:44.7770653Z ++ git cherry -v origin/master
2022-11-23T01:02:44.7784543Z + COMMIT_MESSAGES=
2022-11-23T01:02:44.7784871Z + COMMIT_MESSAGES=
2022-11-23T01:02:44.7785179Z + PR_BODY=
2022-11-23T01:02:44.7785570Z + export COMMIT_MESSAGES=
2022-11-23T01:02:44.7785780Z + COMMIT_MESSAGES=
2022-11-23T01:02:44.7785959Z + export PR_BODY=
2022-11-23T01:02:44.7786115Z + PR_BODY=
2022-11-23T01:02:44.7794182Z +++ nproc --ignore=2
2022-11-23T01:02:44.7804800Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293562 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:02:55.7468012Z + container_name=5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T01:02:55.7468585Z + echo DOCKER_CONTAINER_ID=5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T01:02:55.7471911Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl
2022-11-23T01:02:55.7473762Z + docker exec -t 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5 sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-11-23T01:02:56.1544066Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp37-cp37m-linux_x86_64.whl
2022-11-23T01:02:56.5155970Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (4.3.0)
2022-11-23T01:02:56.5159574Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (1.10.1)
2022-11-23T01:02:56.5162762Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:02:56.5176577Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:02:56.5240827Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.18.5)
2022-11-23T01:02:56.5419112Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:02:57.3076850Z Installing collected packages: torch
2022-11-23T01:03:02.9471604Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:03:03.0099576Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-11-23T01:03:03.0255116Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2022-11-23T01:03:03.0255809Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2022-11-23T01:03:03.0256157Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2022-11-23T01:03:03.0256514Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2022-11-23T01:03:03.0258543Z + BUILD_DIR=build
2022-11-23T01:03:03.0258747Z + BUILD_RENAMED_DIR=build_renamed
2022-11-23T01:03:03.0259043Z + BUILD_BIN_DIR=build/bin
2022-11-23T01:03:03.0261099Z + export VALGRIND=ON
2022-11-23T01:03:03.0261652Z + VALGRIND=ON
2022-11-23T01:03:03.0262196Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]]
2022-11-23T01:03:03.0262546Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]]
2022-11-23T01:03:03.0262781Z ++ realpath build/custom_test_artifacts
2022-11-23T01:03:03.0265385Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-11-23T01:03:03.0273030Z ++ dirname .jenkins/pytorch/test.sh
2022-11-23T01:03:03.0274587Z + source .jenkins/pytorch/common.sh
2022-11-23T01:03:03.0277793Z +++ dirname .jenkins/pytorch/common.sh
2022-11-23T01:03:03.0282488Z ++ source .jenkins/pytorch/common_utils.sh
2022-11-23T01:03:03.0284978Z +++ declare -f -t trap_add
2022-11-23T01:03:03.0292120Z ++ set -ex
2022-11-23T01:03:03.0292532Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-11-23T01:03:03.0292856Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:03:03.0293106Z + echo 'Environment variables'
2022-11-23T01:03:03.0293346Z Environment variables
2022-11-23T01:03:03.0293524Z + env
2022-11-23T01:03:03.0299983Z INSTALLED_DB=yes
2022-11-23T01:03:03.0300894Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:03:03.0301642Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2022-11-23T01:03:03.0302075Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-11-23T01:03:03.0302458Z HOSTNAME=5347e8488c91
2022-11-23T01:03:03.0303113Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_1363f541-c9a2-48de-8508-d66f4c39ec37
2022-11-23T01:03:03.0303418Z GITHUB_ACTION=__self
2022-11-23T01:03:03.0303609Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2022-11-23T01:03:03.0303817Z GITHUB_RUN_NUMBER=67324
2022-11-23T01:03:03.0304007Z TEST_CONFIG=distributed
2022-11-23T01:03:03.0304246Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-11-23T01:03:03.0304670Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2022-11-23T01:03:03.0304885Z GITHUB_REF_TYPE=branch
2022-11-23T01:03:03.0305068Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-11-23T01:03:03.0305318Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:03:03.0305598Z XLA_CUDA=
2022-11-23T01:03:03.0306338Z ***
2022-11-23T01:03:03.0306625Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-11-23T01:03:03.0306932Z GITHUB_ACTIONS=true
2022-11-23T01:03:03.0307236Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:03:03.0307606Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:03:03.0308082Z UCC_HOME=/usr
2022-11-23T01:03:03.0308379Z GITHUB_REF=refs/heads/master
2022-11-23T01:03:03.0308705Z SHARD_NUMBER=1
2022-11-23T01:03:03.0309026Z GITHUB_REF_PROTECTED=true
2022-11-23T01:03:03.0309355Z HOME=/var/lib/jenkins
2022-11-23T01:03:03.0309709Z GITHUB_API_URL=https://api.github.com
2022-11-23T01:03:03.0310128Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2022-11-23T01:03:03.0310483Z INSTALLED_THRIFT=
2022-11-23T01:03:03.0310776Z UCX_COMMIT=
2022-11-23T01:03:03.0311078Z SCCACHE_S3_KEY_PREFIX=pull
2022-11-23T01:03:03.0311404Z NUM_TEST_SHARDS=2
2022-11-23T01:03:03.0311676Z UCX_HOME=/usr
2022-11-23T01:03:03.0312414Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_1363f541-c9a2-48de-8508-d66f4c39ec37
2022-11-23T01:03:03.0312950Z PYTORCH_RETRY_TEST_CASES=1
2022-11-23T01:03:03.0313631Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_1363f541-c9a2-48de-8508-d66f4c39ec37
2022-11-23T01:03:03.0314430Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-11-23T01:03:03.0314915Z GITHUB_EVENT_NAME=push
2022-11-23T01:03:03.0315235Z GITHUB_RUN_ID=3528293562
2022-11-23T01:03:03.0315970Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_1363f541-c9a2-48de-8508-d66f4c39ec37
2022-11-23T01:03:03.0316533Z GITHUB_ACTOR=pytorchmergebot
2022-11-23T01:03:03.0316870Z PR_NUMBER=
2022-11-23T01:03:03.0317157Z DESIRED_CUDA=
2022-11-23T01:03:03.0317453Z GITHUB_RUN_ATTEMPT=1
2022-11-23T01:03:03.0317767Z VALGRIND=ON
2022-11-23T01:03:03.0318118Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-11-23T01:03:03.0318405Z TERM=xterm
2022-11-23T01:03:03.0318580Z INSTALLED_VISION=yes
2022-11-23T01:03:03.0318747Z BRANCH=master
2022-11-23T01:03:03.0318932Z OPENSSL_ROOT_DIR=/opt/openssl
2022-11-23T01:03:03.0319133Z CUDA_PATH=/usr/local/cuda
2022-11-23T01:03:03.0319515Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-11-23T01:03:03.0319827Z GITHUB_SERVER_URL=https://github.com
2022-11-23T01:03:03.0320029Z UCC_COMMIT=
2022-11-23T01:03:03.0320188Z INSTALLED_ANDROID=
2022-11-23T01:03:03.0320360Z SHLVL=1
2022-11-23T01:03:03.0320517Z MAX_JOBS=6
2022-11-23T01:03:03.0320671Z COMMIT_MESSAGES=
2022-11-23T01:03:03.0320854Z GITHUB_REF_NAME=master
2022-11-23T01:03:03.0321185Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:03:03.0321435Z GITHUB_JOB=test
2022-11-23T01:03:03.0321635Z GITHUB_REPOSITORY=pytorch/pytorch
2022-11-23T01:03:03.0321843Z GITHUB_RETENTION_DAYS=90
2022-11-23T01:03:03.0322026Z OPENSSL_DIR=/opt/openssl
2022-11-23T01:03:03.0322224Z GITHUB_ACTION_REPOSITORY=
2022-11-23T01:03:03.0322540Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:03:03.0322814Z GITHUB_BASE_REF=
2022-11-23T01:03:03.0322980Z CI=true
2022-11-23T01:03:03.0323163Z GITHUB_REPOSITORY_OWNER=pytorch
2022-11-23T01:03:03.0323357Z INSTALLED_PROTOBUF=yes
2022-11-23T01:03:03.0323543Z GITHUB_HEAD_REF=
2022-11-23T01:03:03.0323721Z GITHUB_ACTION_REF=
2022-11-23T01:03:03.0323989Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-11-23T01:03:03.0324226Z GITHUB_WORKFLOW=pull
2022-11-23T01:03:03.0324426Z DEBIAN_FRONTEND=noninteractive
2022-11-23T01:03:03.0324948Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_1363f541-c9a2-48de-8508-d66f4c39ec37
2022-11-23T01:03:03.0325213Z PR_BODY=
2022-11-23T01:03:03.0325377Z _=/usr/bin/env
2022-11-23T01:03:03.0325612Z + echo 'Testing pytorch'
2022-11-23T01:03:03.0325786Z Testing pytorch
2022-11-23T01:03:03.0325983Z + export LANG=C.UTF-8
2022-11-23T01:03:03.0326176Z + LANG=C.UTF-8
2022-11-23T01:03:03.0326330Z + PR_NUMBER=
2022-11-23T01:03:03.0326518Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-11-23T01:03:03.0326745Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-11-23T01:03:03.0327004Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-11-23T01:03:03.0327221Z + [[ distributed == \s\l\o\w ]]
2022-11-23T01:03:03.0327542Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]]
2022-11-23T01:03:03.0327812Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-11-23T01:03:03.0328084Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-11-23T01:03:03.0328299Z + [[ distributed == *crossref* ]]
2022-11-23T01:03:03.0328492Z + [[ distributed == *dynamo* ]]
2022-11-23T01:03:03.0328700Z + [[ distributed == *inductor* ]]
2022-11-23T01:03:03.0328954Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-11-23T01:03:03.0329232Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]]
2022-11-23T01:03:03.0329478Z + pip_install --user ninja==1.10.2
2022-11-23T01:03:03.0329769Z + pip install --progress-bar off --user ninja==1.10.2
2022-11-23T01:03:03.4282996Z Collecting ninja==1.10.2
2022-11-23T01:03:03.4671644Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-11-23T01:03:04.1933084Z Installing collected packages: ninja
2022-11-23T01:03:04.2019338Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-11-23T01:03:04.2020052Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T01:03:04.2075098Z Successfully installed ninja-1.10.2
2022-11-23T01:03:04.2728065Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:03:04.2728736Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:03:04.2729367Z + [[ linux-focal-py3.7-gcc7 == *asan* ]]
2022-11-23T01:03:04.2729909Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2022-11-23T01:03:04.2730253Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-11-23T01:03:04.2730498Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-11-23T01:03:04.2735107Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]]
2022-11-23T01:03:04.2747534Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-11-23T01:03:04.2747887Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]]
2022-11-23T01:03:04.2748149Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2022-11-23T01:03:04.2750341Z + cd test
2022-11-23T01:03:04.2750660Z + python -c 'import torch; print(torch.__config__.show())'
2022-11-23T01:03:05.3492186Z PyTorch built with:
2022-11-23T01:03:05.3492627Z   - GCC 7.5
2022-11-23T01:03:05.3492839Z   - C++ Version: 201402
2022-11-23T01:03:05.3493233Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:03:05.3493649Z   - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:03:05.3493938Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:03:05.3494216Z   - LAPACK is enabled (usually provided by MKL)
2022-11-23T01:03:05.3494442Z   - NNPACK is enabled
2022-11-23T01:03:05.3494669Z   - CPU capability usage: AVX2
2022-11-23T01:03:05.3497392Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-11-23T01:03:05.3499296Z 
2022-11-23T01:03:05.5300677Z + cd test
2022-11-23T01:03:05.5301142Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-11-23T01:03:06.5820505Z ATen/Parallel:
2022-11-23T01:03:06.5832050Z 	at::get_num_threads() : 4
2022-11-23T01:03:06.5832301Z 	at::get_num_interop_threads() : 4
2022-11-23T01:03:06.5832544Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:03:06.5832733Z 	omp_get_max_threads() : 4
2022-11-23T01:03:06.5833239Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:03:06.5833528Z 	mkl_get_max_threads() : 4
2022-11-23T01:03:06.5833832Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:03:06.5834102Z std::thread::hardware_concurrency() : 8
2022-11-23T01:03:06.5834315Z Environment variables:
2022-11-23T01:03:06.5834509Z 	OMP_NUM_THREADS : [not set]
2022-11-23T01:03:06.5834693Z 	MKL_NUM_THREADS : [not set]
2022-11-23T01:03:06.5834894Z ATen parallel backend: OpenMP
2022-11-23T01:03:06.5835018Z 
2022-11-23T01:03:06.7624102Z + [[ distributed == *backward* ]]
2022-11-23T01:03:06.7624492Z + [[ distributed == *xla* ]]
2022-11-23T01:03:06.7624865Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-11-23T01:03:06.7625234Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-11-23T01:03:06.7625452Z + [[ distributed == distributed ]]
2022-11-23T01:03:06.7625649Z + install_filelock
2022-11-23T01:03:06.7625836Z + pip_install filelock
2022-11-23T01:03:06.7626078Z + pip install --progress-bar off filelock
2022-11-23T01:03:07.1416403Z Collecting filelock
2022-11-23T01:03:07.1586752Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T01:03:07.8861430Z Installing collected packages: filelock
2022-11-23T01:03:07.9148787Z Successfully installed filelock-3.8.0
2022-11-23T01:03:07.9782174Z + install_triton
2022-11-23T01:03:07.9782561Z + local commit
2022-11-23T01:03:07.9782792Z + [[ distributed == *rocm* ]]
2022-11-23T01:03:07.9785035Z ++ get_pinned_commit triton
2022-11-23T01:03:07.9785327Z ++ cat .github/ci_commit_pins/triton.txt
2022-11-23T01:03:07.9795975Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:03:07.9796975Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:03:07.9797524Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:03:08.3180410Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:03:08.3185990Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-kmb0umhe
2022-11-23T01:03:08.3217578Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-kmb0umhe
2022-11-23T01:03:08.9718818Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2022-11-23T01:03:08.9752674Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:03:09.2906246Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:03:09.4577658Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:03:09.4579405Z   Running command git submodule update --init --recursive -q
2022-11-23T01:03:09.9459932Z   Preparing metadata (setup.py) ... [?25l- done
2022-11-23T01:03:10.1085192Z [?25hCollecting cmake
2022-11-23T01:03:10.1288872Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2022-11-23T01:03:10.4280390Z Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (3.8.0)
2022-11-23T01:03:10.4283645Z Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385)
2022-11-23T01:03:10.4496690Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (2.6.3)
2022-11-23T01:03:10.4500108Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (1.10.1)
2022-11-23T01:03:10.4503824Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (4.3.0)
2022-11-23T01:03:10.4675515Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2022-11-23T01:03:10.4738551Z Building wheels for collected packages: triton
2022-11-23T01:03:53.9269853Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ done
2022-11-23T01:03:53.9640742Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp37-cp37m-linux_x86_64.whl size=15370740 sha256=495e8e4479bd4e3fe420d062f8ce2187c5fc2fa4bef84ead781b05222bcab0e3
2022-11-23T01:03:53.9641746Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/aa/c5/3a/263a5ba27ed44796dab5f19b43135afb68beabac5d137652f7
2022-11-23T01:03:53.9666162Z Successfully built triton
2022-11-23T01:03:54.7009926Z Installing collected packages: cmake, triton
2022-11-23T01:03:55.8843268Z Successfully installed cmake-3.25.0 triton-2.0.0
2022-11-23T01:03:55.9800049Z + pip_install --user jinja2
2022-11-23T01:03:55.9800350Z + pip install --progress-bar off --user jinja2
2022-11-23T01:03:56.3617175Z Collecting jinja2
2022-11-23T01:03:56.3809000Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T01:03:56.3924606Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.7/site-packages (from jinja2) (2.1.1)
2022-11-23T01:03:57.1328444Z Installing collected packages: jinja2
2022-11-23T01:03:57.2084430Z Successfully installed jinja2-3.1.2
2022-11-23T01:03:57.2709884Z + test_distributed
2022-11-23T01:03:57.2710542Z + echo 'Testing distributed python tests'
2022-11-23T01:03:57.2710783Z Testing distributed python tests
2022-11-23T01:03:57.2711712Z + python test/run_test.py --distributed-tests --shard 1 2 --verbose
2022-11-23T01:03:58.7305251Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-11-23T01:03:58.8518326Z Ignoring disabled issues:  []
2022-11-23T01:03:58.8788118Z Found test time stats from artifacts
2022-11-23T01:03:58.8799725Z Selected tests:
2022-11-23T01:03:58.8800243Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:03:58.8800645Z  distributed/test_distributed_spawn
2022-11-23T01:03:58.8800988Z  distributed/pipeline/sync/test_worker
2022-11-23T01:03:58.8801207Z  distributed/pipeline/sync/test_stream
2022-11-23T01:03:58.8801438Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:03:58.8801767Z  distributed/pipeline/sync/test_microbatch
2022-11-23T01:03:58.8802151Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:03:58.8802433Z  distributed/pipeline/sync/test_copy
2022-11-23T01:03:58.8802718Z  distributed/pipeline/sync/test_bugs
2022-11-23T01:03:58.8803267Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:03:58.8803694Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:03:58.8804256Z  distributed/pipeline/sync/skip/test_leak
2022-11-23T01:03:58.8804684Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:03:58.8805135Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:03:58.8805469Z  distributed/elastic/timer/api_test
2022-11-23T01:03:58.8805688Z  distributed/checkpoint/test_traverse
2022-11-23T01:03:58.8805894Z  distributed/_shard/test_sharder
2022-11-23T01:03:58.8806130Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T01:03:58.8806363Z  distributed/test_c10d_spawn_ucc
2022-11-23T01:03:58.8806583Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T01:03:58.8806929Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T01:03:58.8807201Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T01:03:58.8809363Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T01:03:58.8809820Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T01:03:58.8810096Z  distributed/test_c10d_spawn_nccl
2022-11-23T01:03:58.8810371Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T01:03:58.8810626Z  distributed/_shard/test_partial_tensor
2022-11-23T01:03:58.8810869Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:03:58.8811137Z  distributed/test_data_parallel
2022-11-23T01:03:58.8811589Z  distributed/test_launcher
2022-11-23T01:03:58.8811937Z  distributed/_composable/test_contract
2022-11-23T01:03:58.8812280Z  distributed/elastic/utils/logging_test
2022-11-23T01:03:58.8812486Z  distributed/fsdp/test_utils
2022-11-23T01:03:58.8812697Z  distributed/test_multi_threaded_pg
2022-11-23T01:03:58.8812981Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:03:58.8813226Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T01:03:58.8813467Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:03:58.8813773Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:03:58.8814003Z  distributed/rpc/test_share_memory
2022-11-23T01:03:58.8814224Z  distributed/elastic/utils/distributed_test
2022-11-23T01:03:58.8814502Z  distributed/fsdp/test_fsdp_input
2022-11-23T01:03:58.8814737Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:03:58.8815209Z  distributed/elastic/timer/local_timer_test
2022-11-23T01:03:58.8815646Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:03:58.8818730Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:03:58.8819113Z  distributed/_tensor/parallel/test_tp_examples
2022-11-23T01:03:58.8819369Z  distributed/test_c10d_object_collectives
2022-11-23T01:03:58.8819602Z  distributed/_composable/test_fully_shard
2022-11-23T01:03:58.8819880Z  distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T01:03:58.8820127Z  distributed/fsdp/test_fsdp_tp_integration
2022-11-23T01:03:58.8820369Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:03:58.8820623Z  distributed/test_c10d_common
2022-11-23T01:03:58.8820853Z  distributed/fsdp/test_fsdp_flatten_params
2022-11-23T01:03:58.8821078Z  distributed/algorithms/test_join
2022-11-23T01:03:58.8821291Z  distributed/fsdp/test_fsdp_comm
2022-11-23T01:03:58.8821556Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:03:58.8821780Z  distributed/_tensor/test_redistribute
2022-11-23T01:03:58.8821982Z  distributed/test_pg_wrapper
2022-11-23T01:03:58.8822243Z  distributed/_tensor/test_common_rules
2022-11-23T01:03:58.8822463Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:03:58.8822680Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:03:58.8822923Z  distributed/_tensor/test_dtensor
2022-11-23T01:03:58.8823148Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:03:58.8823358Z  distributed/fsdp/test_wrap
2022-11-23T01:03:58.8823548Z  distributed/test_c10d_nccl
2022-11-23T01:03:58.8823794Z  distributed/test_c10d_pypg
2022-11-23T01:03:58.8824006Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:03:58.8900468Z Prioritized test from test file changes.
2022-11-23T01:03:58.8900973Z reordering tests for PR:
2022-11-23T01:03:58.8901351Z prioritized: []
2022-11-23T01:03:58.8906347Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/pipeline/sync/test_worker', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/optim/test_apply_optimizer_in_backward', 'distributed/elastic/timer/api_test', 'distributed/checkpoint/test_traverse', 'distributed/_shard/test_sharder', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/test_c10d_spawn_ucc', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/test_c10d_spawn_nccl', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/test_data_parallel', 'distributed/test_launcher', 'distributed/_composable/test_contract', 'distributed/elastic/utils/logging_test', 'distributed/fsdp/test_utils', 'distributed/test_multi_threaded_pg', 'distributed/fsdp/test_fsdp_traversal', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/rpc/test_share_memory', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/timer/local_timer_test', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_apply', 'distributed/_tensor/parallel/test_tp_examples', 'distributed/test_c10d_object_collectives', 'distributed/_composable/test_fully_shard', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/elastic/timer/local_timer_example', 'distributed/test_c10d_common', 'distributed/fsdp/test_fsdp_flatten_params', 'distributed/algorithms/test_join', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/_tensor/test_redistribute', 'distributed/test_pg_wrapper', 'distributed/_tensor/test_common_rules', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_misc', 'distributed/_tensor/test_dtensor', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_wrap', 'distributed/test_c10d_nccl', 'distributed/test_c10d_pypg', 'distributed/rpc/test_tensorpipe_agent']
2022-11-23T01:03:58.8909416Z 
2022-11-23T01:03:58.8909838Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:03:58.9117895Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:03:58.9313565Z parallel (file granularity) tests:
2022-11-23T01:03:58.9313888Z  
2022-11-23T01:03:58.9314206Z serial (file granularity) tests:
2022-11-23T01:03:58.9314669Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:03:58.9315098Z  distributed/test_distributed_spawn
2022-11-23T01:03:58.9315461Z  distributed/pipeline/sync/test_worker
2022-11-23T01:03:58.9315875Z  distributed/pipeline/sync/test_stream
2022-11-23T01:03:58.9316233Z  distributed/pipeline/sync/test_pipe
2022-11-23T01:03:58.9316619Z  distributed/pipeline/sync/test_microbatch
2022-11-23T01:03:58.9317071Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:03:58.9317439Z  distributed/pipeline/sync/test_copy
2022-11-23T01:03:58.9317798Z  distributed/pipeline/sync/test_bugs
2022-11-23T01:03:58.9318463Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:03:58.9318781Z  distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:03:58.9319026Z  distributed/pipeline/sync/skip/test_leak
2022-11-23T01:03:58.9319262Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:03:58.9319512Z  distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:03:58.9319736Z  distributed/elastic/timer/api_test
2022-11-23T01:03:58.9319960Z  distributed/checkpoint/test_traverse
2022-11-23T01:03:58.9320176Z  distributed/_shard/test_sharder
2022-11-23T01:03:58.9320399Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T01:03:58.9320687Z  distributed/test_c10d_spawn_ucc
2022-11-23T01:03:58.9320921Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T01:03:58.9321177Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T01:03:58.9321444Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T01:03:58.9321716Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T01:03:58.9321963Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T01:03:58.9322191Z  distributed/test_c10d_spawn_nccl
2022-11-23T01:03:58.9322431Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T01:03:58.9322659Z  distributed/_shard/test_partial_tensor
2022-11-23T01:03:58.9322901Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:03:58.9323130Z  distributed/test_data_parallel
2022-11-23T01:03:58.9323330Z  distributed/test_launcher
2022-11-23T01:03:58.9323527Z  distributed/_composable/test_contract
2022-11-23T01:03:58.9323758Z  distributed/elastic/utils/logging_test
2022-11-23T01:03:58.9323974Z  distributed/fsdp/test_utils
2022-11-23T01:03:58.9324171Z  distributed/test_multi_threaded_pg
2022-11-23T01:03:58.9324392Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:03:58.9324638Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T01:03:58.9324877Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:03:58.9325124Z  distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:03:58.9325353Z  distributed/rpc/test_share_memory
2022-11-23T01:03:58.9325569Z  distributed/elastic/utils/distributed_test
2022-11-23T01:03:58.9325797Z  distributed/fsdp/test_fsdp_input
2022-11-23T01:03:58.9326028Z  distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:03:58.9326254Z  distributed/elastic/timer/local_timer_test
2022-11-23T01:03:58.9326503Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:03:58.9326739Z  distributed/fsdp/test_fsdp_apply
2022-11-23T01:03:58.9326957Z  distributed/_tensor/parallel/test_tp_examples
2022-11-23T01:03:58.9327198Z  distributed/test_c10d_object_collectives
2022-11-23T01:03:58.9327449Z  distributed/_composable/test_fully_shard
2022-11-23T01:03:58.9327684Z  distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T01:03:58.9327908Z  distributed/fsdp/test_fsdp_tp_integration
2022-11-23T01:03:58.9328147Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:03:58.9328369Z  distributed/test_c10d_common
2022-11-23T01:03:58.9328574Z  distributed/fsdp/test_fsdp_flatten_params
2022-11-23T01:03:58.9328797Z  distributed/algorithms/test_join
2022-11-23T01:03:58.9329010Z  distributed/fsdp/test_fsdp_comm
2022-11-23T01:03:58.9329211Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:03:58.9329436Z  distributed/_tensor/test_redistribute
2022-11-23T01:03:58.9329649Z  distributed/test_pg_wrapper
2022-11-23T01:03:58.9329846Z  distributed/_tensor/test_common_rules
2022-11-23T01:03:58.9330065Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:03:58.9330279Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:03:58.9330475Z  distributed/_tensor/test_dtensor
2022-11-23T01:03:58.9330687Z  distributed/_tensor/test_tensor_ops
2022-11-23T01:03:58.9330898Z  distributed/fsdp/test_wrap
2022-11-23T01:03:58.9331097Z  distributed/test_c10d_nccl
2022-11-23T01:03:58.9331277Z  distributed/test_c10d_pypg
2022-11-23T01:03:58.9331586Z  distributed/rpc/test_tensorpipe_agent
2022-11-23T01:04:00.4285665Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-11-23T01:04:00.4286290Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-11-23T01:04:00.5494713Z Ignoring disabled issues:  []
2022-11-23T01:04:00.5666101Z Ignoring disabled issues:  []
2022-11-23T01:04:00.9121283Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:04:00.911734]
2022-11-23T01:04:00.9172377Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:04:00.9173126Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2022-11-23T01:04:00.9190199Z Running distributed tests for the test backend with env init_method in shard 1 of 2
2022-11-23T01:04:00.9195854Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:00.919364]
2022-11-23T01:04:02.3256705Z 
2022-11-23T01:04:02.3257365Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:04:02.3262331Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_9t_2_8lo)
2022-11-23T01:04:02.3262804Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:02.3262950Z 
2022-11-23T01:04:02.3263177Z ##[endgroup]
2022-11-23T01:04:02.3263825Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_9t_2_8lo)
2022-11-23T01:04:02.3264158Z 
2022-11-23T01:04:02.3305146Z Running distributed tests for the test backend with file init_method in shard 1 of 2
2022-11-23T01:04:02.3309420Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:02.330740]
2022-11-23T01:04:03.7371947Z 
2022-11-23T01:04:03.7372673Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:04:03.7373835Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_zwii_pmw)
2022-11-23T01:04:03.7374280Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:03.7374421Z 
2022-11-23T01:04:03.7374635Z ##[endgroup]
2022-11-23T01:04:03.7375424Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_zwii_pmw)
2022-11-23T01:04:03.7375750Z 
2022-11-23T01:04:03.7389502Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2
2022-11-23T01:04:03.7393884Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:03.739190]
2022-11-23T01:04:18.3803032Z 
2022-11-23T01:04:18.3803661Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:04:18.3805168Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_faypoz87)
2022-11-23T01:04:18.3806127Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:04:18.3806988Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3807248Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3807515Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3807776Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3808049Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3808314Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:18.3808567Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:18.3809085Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3809665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3809998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3810432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3810780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3810933Z 
2022-11-23T01:04:18.3811005Z Running tests...
2022-11-23T01:04:18.3811308Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3811755Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 730
2022-11-23T01:04:18.3812131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 731
2022-11-23T01:04:18.3812459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 732
2022-11-23T01:04:18.3813210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3813543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3813963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3814309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3814634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:04:18.3815342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3815673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3816108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3816454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3816763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:04:18.3817222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3817549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3817975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3818303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3818623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:04:18.3818983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:04:18.3819340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:04:18.3819833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3820229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:04:18.3820941Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3821454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3821746Z ok (2.461s)
2022-11-23T01:04:18.3821851Z 
2022-11-23T01:04:18.3822051Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3822293Z Ran 1 test in 2.461s
2022-11-23T01:04:18.3822392Z 
2022-11-23T01:04:18.3822454Z OK
2022-11-23T01:04:18.3822590Z 
2022-11-23T01:04:18.3822674Z Generating XML reports...
2022-11-23T01:04:18.3823179Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010406.xml
2022-11-23T01:04:18.3823755Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3824260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3824590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3825022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3825353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3825515Z 
2022-11-23T01:04:18.3825587Z Running tests...
2022-11-23T01:04:18.3825889Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3826263Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 783
2022-11-23T01:04:18.3826645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 784
2022-11-23T01:04:18.3826969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 785
2022-11-23T01:04:18.3827415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3827730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3828257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3828600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3829036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:04:18.3829534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3829857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3830285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3830617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3830941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:04:18.3831400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3831724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3832134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3832541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3832868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:04:18.3833311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:04:18.3833714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:04:18.3834278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3834677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:04:18.3835153Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3835669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:18.3835994Z ok (1.974s)
2022-11-23T01:04:18.3836095Z 
2022-11-23T01:04:18.3836295Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3836519Z Ran 1 test in 1.975s
2022-11-23T01:04:18.3836630Z 
2022-11-23T01:04:18.3836689Z OK
2022-11-23T01:04:18.3836778Z 
2022-11-23T01:04:18.3836860Z Generating XML reports...
2022-11-23T01:04:18.3837351Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010410.xml
2022-11-23T01:04:18.3837935Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3838440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3838767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3839180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3839526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3839686Z 
2022-11-23T01:04:18.3839758Z Running tests...
2022-11-23T01:04:18.3840041Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3840393Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:18.3840597Z 
2022-11-23T01:04:18.3840789Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3841026Z Ran 1 test in 0.001s
2022-11-23T01:04:18.3841124Z 
2022-11-23T01:04:18.3841195Z OK (skipped=1)
2022-11-23T01:04:18.3841299Z 
2022-11-23T01:04:18.3841381Z Generating XML reports...
2022-11-23T01:04:18.3841878Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010413.xml
2022-11-23T01:04:18.3842442Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3842949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3843276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3843703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3844030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3844194Z 
2022-11-23T01:04:18.3844265Z Running tests...
2022-11-23T01:04:18.3844567Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3844915Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:18.3845107Z 
2022-11-23T01:04:18.3845338Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3845575Z Ran 1 test in 0.001s
2022-11-23T01:04:18.3845683Z 
2022-11-23T01:04:18.3845753Z OK (skipped=1)
2022-11-23T01:04:18.3845859Z 
2022-11-23T01:04:18.3845929Z Generating XML reports...
2022-11-23T01:04:18.3846421Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010415.xml
2022-11-23T01:04:18.3847038Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3847541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3847852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3848276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3848618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3848813Z 
2022-11-23T01:04:18.3848885Z Running tests...
2022-11-23T01:04:18.3849171Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3849525Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:18.3849739Z 
2022-11-23T01:04:18.3849933Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3850157Z Ran 1 test in 0.001s
2022-11-23T01:04:18.3850269Z 
2022-11-23T01:04:18.3850340Z OK (skipped=1)
2022-11-23T01:04:18.3850444Z 
2022-11-23T01:04:18.3850525Z Generating XML reports...
2022-11-23T01:04:18.3851021Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010416.xml
2022-11-23T01:04:18.3851662Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:18.3852175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:18.3852509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:18.3852926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:18.3853270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:18.3853436Z 
2022-11-23T01:04:18.3853508Z Running tests...
2022-11-23T01:04:18.3853806Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3854147Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:18.3854358Z 
2022-11-23T01:04:18.3854552Z ----------------------------------------------------------------------
2022-11-23T01:04:18.3854787Z Ran 1 test in 0.001s
2022-11-23T01:04:18.3854998Z 
2022-11-23T01:04:18.3855063Z OK (skipped=1)
2022-11-23T01:04:18.3855199Z 
2022-11-23T01:04:18.3855282Z Generating XML reports...
2022-11-23T01:04:18.3855786Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010417.xml
2022-11-23T01:04:18.3856086Z 
2022-11-23T01:04:18.3856390Z ##[endgroup]
2022-11-23T01:04:18.3856938Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_faypoz87)
2022-11-23T01:04:18.3857265Z 
2022-11-23T01:04:18.3857435Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2
2022-11-23T01:04:18.3858033Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:18.383193]
2022-11-23T01:04:32.4697558Z 
2022-11-23T01:04:32.4700215Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:04:32.4701176Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_d2h6xov2)
2022-11-23T01:04:32.4702237Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:04:32.4702900Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4703173Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4703438Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4703750Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4704023Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4704300Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:04:32.4704554Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:32.4705136Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4705661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4705994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4706411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4706762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4706928Z 
2022-11-23T01:04:32.4707001Z Running tests...
2022-11-23T01:04:32.4707308Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4707681Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 886
2022-11-23T01:04:32.4708066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 887
2022-11-23T01:04:32.4708394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 888
2022-11-23T01:04:32.4708829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4709158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4709585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4709933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4710241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:04:32.4710732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4711058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4711489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4711831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4712154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:04:32.4712612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4712924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4713345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4713690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4714011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:04:32.4714359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:04:32.4714775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:04:32.4715140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:04:32.4715622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4716142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4716658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4716980Z ok (1.974s)
2022-11-23T01:04:32.4717080Z 
2022-11-23T01:04:32.4717264Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4717507Z Ran 1 test in 1.974s
2022-11-23T01:04:32.4717620Z 
2022-11-23T01:04:32.4717679Z OK
2022-11-23T01:04:32.4717770Z 
2022-11-23T01:04:32.4717843Z Generating XML reports...
2022-11-23T01:04:32.4718348Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010420.xml
2022-11-23T01:04:32.4718936Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4719446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4719758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4720182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4720529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4720692Z 
2022-11-23T01:04:32.4720764Z Running tests...
2022-11-23T01:04:32.4721051Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4721443Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 939
2022-11-23T01:04:32.4721869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 940
2022-11-23T01:04:32.4722179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 941
2022-11-23T01:04:32.4722676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4723006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4723439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4723767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4724091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:04:32.4724549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4724860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4725286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4725631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4725950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:04:32.4726394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4726720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4727144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4727484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4727849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:04:32.4728211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:04:32.4728574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:04:32.4728923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:04:32.4729414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4729964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4730480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:04:32.4730748Z ok (1.972s)
2022-11-23T01:04:32.4730852Z 
2022-11-23T01:04:32.4731048Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4731287Z Ran 1 test in 1.973s
2022-11-23T01:04:32.4731400Z 
2022-11-23T01:04:32.4731460Z OK
2022-11-23T01:04:32.4731631Z 
2022-11-23T01:04:32.4731719Z Generating XML reports...
2022-11-23T01:04:32.4732225Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010424.xml
2022-11-23T01:04:32.4732807Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4733302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4733631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4734057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4734404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4734555Z 
2022-11-23T01:04:32.4734626Z Running tests...
2022-11-23T01:04:32.4735107Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4735465Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:32.4735672Z 
2022-11-23T01:04:32.4735856Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4736097Z Ran 1 test in 0.001s
2022-11-23T01:04:32.4736210Z 
2022-11-23T01:04:32.4736282Z OK (skipped=1)
2022-11-23T01:04:32.4736389Z 
2022-11-23T01:04:32.4736471Z Generating XML reports...
2022-11-23T01:04:32.4736957Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010427.xml
2022-11-23T01:04:32.4737550Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4738054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4738384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4738796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4739140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4739300Z 
2022-11-23T01:04:32.4739371Z Running tests...
2022-11-23T01:04:32.4739658Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4740009Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:32.4740211Z 
2022-11-23T01:04:32.4740404Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4740639Z Ran 1 test in 0.001s
2022-11-23T01:04:32.4740812Z 
2022-11-23T01:04:32.4740885Z OK (skipped=1)
2022-11-23T01:04:32.4740991Z 
2022-11-23T01:04:32.4741074Z Generating XML reports...
2022-11-23T01:04:32.4741570Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010429.xml
2022-11-23T01:04:32.4742140Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4742641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4743012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4743437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4743768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4743933Z 
2022-11-23T01:04:32.4744009Z Running tests...
2022-11-23T01:04:32.4744310Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4744654Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:32.4744866Z 
2022-11-23T01:04:32.4745063Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4745297Z Ran 1 test in 0.001s
2022-11-23T01:04:32.4745408Z 
2022-11-23T01:04:32.4745479Z OK (skipped=1)
2022-11-23T01:04:32.4745586Z 
2022-11-23T01:04:32.4745657Z Generating XML reports...
2022-11-23T01:04:32.4746156Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010430.xml
2022-11-23T01:04:32.4746741Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:04:32.4747244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:04:32.4747555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:04:32.4747980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:04:32.4748321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:04:32.4748482Z 
2022-11-23T01:04:32.4748542Z Running tests...
2022-11-23T01:04:32.4748838Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4749192Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:04:32.4749408Z 
2022-11-23T01:04:32.4749599Z ----------------------------------------------------------------------
2022-11-23T01:04:32.4749822Z Ran 1 test in 0.001s
2022-11-23T01:04:32.4749932Z 
2022-11-23T01:04:32.4750002Z OK (skipped=1)
2022-11-23T01:04:32.4750107Z 
2022-11-23T01:04:32.4750192Z Generating XML reports...
2022-11-23T01:04:32.4750676Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010431.xml
2022-11-23T01:04:32.4750974Z 
2022-11-23T01:04:32.4751265Z ##[endgroup]
2022-11-23T01:04:32.4751808Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_d2h6xov2)
2022-11-23T01:04:32.4752134Z 
2022-11-23T01:04:32.4752346Z Running distributed/test_distributed_spawn ... [2022-11-23 01:04:32.472079]
2022-11-23T01:04:32.4774071Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:04:32.4774519Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1}
2022-11-23T01:04:32.4776050Z Running distributed tests for the test backend with env init_method in shard 1 of 2
2022-11-23T01:04:32.4781511Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:32.477974]
2022-11-23T01:04:34.0731696Z 
2022-11-23T01:04:34.0732180Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:04:34.0732912Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a5rozu93)
2022-11-23T01:04:34.0733287Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:34.0733429Z 
2022-11-23T01:04:34.0733849Z ##[endgroup]
2022-11-23T01:04:34.0734397Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a5rozu93)
2022-11-23T01:04:34.0734656Z 
2022-11-23T01:04:34.0750127Z Running distributed tests for the test backend with file init_method in shard 1 of 2
2022-11-23T01:04:34.0754798Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:34.075286]
2022-11-23T01:04:35.6584725Z 
2022-11-23T01:04:35.6585516Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:04:35.6586279Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_d1u9m29r)
2022-11-23T01:04:35.6586632Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:04:35.6586779Z 
2022-11-23T01:04:35.6586995Z ##[endgroup]
2022-11-23T01:04:35.6587582Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_d1u9m29r)
2022-11-23T01:04:35.6587840Z 
2022-11-23T01:04:35.6628959Z Running distributed tests for the gloo backend with env init_method in shard 1 of 2
2022-11-23T01:04:35.6633345Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:04:35.663135]
2022-11-23T01:19:07.5535668Z 
2022-11-23T01:19:07.5536079Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:19:07.5538761Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fe0m1lpv)
2022-11-23T01:19:07.5546588Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:07.5593878Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:19:07.5635880Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5636468Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5636951Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5637432Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5637940Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5638484Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5736498Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5736989Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5737602Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5738036Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5738463Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5738892Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5739437Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5740165Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5740747Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5741110Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5741537Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5742058Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5742482Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5742948Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5743641Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5744052Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5744429Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5745112Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5745694Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5746196Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5746494Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5746804Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5747106Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5747385Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5747720Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5748040Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5748325Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5748608Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5748924Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5749288Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5749784Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5750260Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5750680Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5750973Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5751263Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5751589Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5751917Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5752248Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5752553Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5752863Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5753179Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5753483Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5753788Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5754110Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5754435Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5754817Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5755342Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5755935Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5756321Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5756619Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5756925Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5757322Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5757810Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5758255Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5758730Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5759272Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5759772Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5760200Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5760654Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5761265Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5761830Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5762352Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5762882Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5763392Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5763860Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5764136Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5764486Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5765017Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5765444Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5765881Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5766326Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5766766Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5767236Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5767696Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5768172Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5768613Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5769082Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5769586Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5770662Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5771181Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5771713Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5772259Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5772785Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5773315Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5773813Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5774304Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5774807Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5775505Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5776040Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5776575Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5777107Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5777624Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5778101Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5778549Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5778989Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5779403Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5779806Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5780255Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5780721Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5781170Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5781635Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5782233Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5782709Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5783156Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5783622Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5784113Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5784586Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5785070Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5785625Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5786423Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5786888Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5787372Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5787900Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5788624Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5789147Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5789676Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5848868Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5849527Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5850125Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5850668Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5851238Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5851867Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5852571Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5853080Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5853643Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5854180Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5854750Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5855534Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5856135Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5856632Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5861698Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5891044Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5917320Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5917932Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5918489Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5919063Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5919670Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5920272Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5920837Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5921408Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5922054Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5922731Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5923661Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5924533Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5925391Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5926217Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5926913Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5927369Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5927826Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5928265Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5928670Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5929038Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5929371Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5929645Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5930026Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5930328Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5930612Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5930925Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5931248Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5931577Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5931926Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5932232Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5932523Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5932816Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5933136Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5933453Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5933746Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5934052Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5934367Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5934685Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5935109Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5935411Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5935711Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5936008Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5936305Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5936611Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5936937Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5937321Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5937616Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5937901Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5938183Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5938491Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5938798Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5939067Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5939376Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5939647Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5939912Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5940188Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5940461Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5940744Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5941011Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5941280Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5941541Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5941804Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5942093Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5942383Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5942645Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5942896Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5943174Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5943469Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5943764Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5944098Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5944431Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5944721Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5945031Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5945351Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5945651Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5945962Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5946269Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5946578Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5946866Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5947166Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5947454Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5947732Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5948075Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5948439Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5948785Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5949105Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5949445Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5949780Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5950111Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5950421Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5950735Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5951055Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5951375Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5951730Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5952136Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5952478Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5952789Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5953082Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5953379Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5953665Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5953948Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5954225Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5954498Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5954783Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5955054Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5955310Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5955585Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5955863Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5956152Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5956436Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5956711Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5956983Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5957255Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5957536Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5957804Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5958090Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5958367Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5958643Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5958916Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5959206Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5959486Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5959771Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5960031Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5960309Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5960623Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5960940Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5961259Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5961552Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5961852Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5962155Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5962460Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5962752Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5963075Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5963402Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5963708Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5963991Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5964293Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5964588Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5964907Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5965203Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5965539Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5965869Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.5966361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.5966849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5967185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5967621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5967957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5968125Z 
2022-11-23T01:19:07.5968198Z Running tests...
2022-11-23T01:19:07.5968507Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5968957Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1063
2022-11-23T01:19:07.5969415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1064
2022-11-23T01:19:07.5969809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1065
2022-11-23T01:19:07.5970267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5970583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5971012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5971358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5971700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.5972160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5972487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5972915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5973245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5973586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.5974060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5974388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5974803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5975273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5975610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.5976186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5976695Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5977212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5977600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.5977935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.5978337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.5978619Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:19:07.5978755Z 
2022-11-23T01:19:07.5978957Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5979186Z Ran 1 test in 2.174s
2022-11-23T01:19:07.5979299Z 
2022-11-23T01:19:07.5979371Z OK (skipped=1)
2022-11-23T01:19:07.5979479Z 
2022-11-23T01:19:07.5979563Z Generating XML reports...
2022-11-23T01:19:07.5980011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010438.xml
2022-11-23T01:19:07.5980508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.5980971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5981306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5981723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5982071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5982235Z 
2022-11-23T01:19:07.5982309Z Running tests...
2022-11-23T01:19:07.5982599Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5982975Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-11-23T01:19:07.5983200Z 
2022-11-23T01:19:07.5983392Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5983627Z Ran 1 test in 0.003s
2022-11-23T01:19:07.5983739Z 
2022-11-23T01:19:07.5983798Z OK (skipped=1)
2022-11-23T01:19:07.5983905Z 
2022-11-23T01:19:07.5983991Z Generating XML reports...
2022-11-23T01:19:07.5984455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010442.xml
2022-11-23T01:19:07.5984950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.5985397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5985728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5986155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5986485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5986649Z 
2022-11-23T01:19:07.5986722Z Running tests...
2022-11-23T01:19:07.5987021Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5987417Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1126
2022-11-23T01:19:07.5987806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1127
2022-11-23T01:19:07.5988136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1128
2022-11-23T01:19:07.5988617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5988930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5989358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5989706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5990041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.5990498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5990867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5991297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5991638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5991960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.5992426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5992752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5993165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.5993507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.5993845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.5994342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5994850Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5995369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.5995755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.5996100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.5996431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.5996678Z ok (2.178s)
2022-11-23T01:19:07.5996780Z 
2022-11-23T01:19:07.5996982Z ----------------------------------------------------------------------
2022-11-23T01:19:07.5997211Z Ran 1 test in 2.178s
2022-11-23T01:19:07.5997322Z 
2022-11-23T01:19:07.5997383Z OK
2022-11-23T01:19:07.5997477Z 
2022-11-23T01:19:07.5997562Z Generating XML reports...
2022-11-23T01:19:07.5998021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010443.xml
2022-11-23T01:19:07.5998509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.5998972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.5999305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.5999721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6000067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6000233Z 
2022-11-23T01:19:07.6000305Z Running tests...
2022-11-23T01:19:07.6000605Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6001504Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.263s)
2022-11-23T01:19:07.6001993Z 
2022-11-23T01:19:07.6002173Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6002410Z Ran 1 test in 0.263s
2022-11-23T01:19:07.6002522Z 
2022-11-23T01:19:07.6002595Z OK (skipped=1)
2022-11-23T01:19:07.6002703Z 
2022-11-23T01:19:07.6002773Z Generating XML reports...
2022-11-23T01:19:07.6003227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010447.xml
2022-11-23T01:19:07.6003753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6004217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6004534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6004960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6005305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6005469Z 
2022-11-23T01:19:07.6005540Z Running tests...
2022-11-23T01:19:07.6005827Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6006246Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1189
2022-11-23T01:19:07.6006675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1190
2022-11-23T01:19:07.6006992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1191
2022-11-23T01:19:07.6007439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6007770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6008199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6008533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6008871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6009337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6009653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6010161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6010510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6010845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6011305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6011633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6012055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6012396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6012715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6013210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6013725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6014259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6014648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6015149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iu13vj6
2022-11-23T01:19:07.6015551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iu13vj6/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6015912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6016262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6016684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xnds_cj
2022-11-23T01:19:07.6017073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4_lcs5z
2022-11-23T01:19:07.6017457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xnds_cj/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6017860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4_lcs5z/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6018240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6018582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6018941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6019199Z ok (2.379s)
2022-11-23T01:19:07.6019303Z 
2022-11-23T01:19:07.6019506Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6019735Z Ran 1 test in 2.379s
2022-11-23T01:19:07.6019847Z 
2022-11-23T01:19:07.6019909Z OK
2022-11-23T01:19:07.6020001Z 
2022-11-23T01:19:07.6020085Z Generating XML reports...
2022-11-23T01:19:07.6020537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010449.xml
2022-11-23T01:19:07.6021035Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6021496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6021829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6022244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6022589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6022754Z 
2022-11-23T01:19:07.6022827Z Running tests...
2022-11-23T01:19:07.6023114Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6023545Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1242
2022-11-23T01:19:07.6023979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1243
2022-11-23T01:19:07.6024307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1244
2022-11-23T01:19:07.6024745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6025074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6025500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6025848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6026170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6026639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6026961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6027410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6027753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6028090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6028557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6028867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6029323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6029667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6029987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6030481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6031006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6031524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6031896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6032243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6032615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p4wmtlk
2022-11-23T01:19:07.6033017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p4wmtlk/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6033404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rler92l
2022-11-23T01:19:07.6033801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rler92l/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6034172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6034541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1hys8a9
2022-11-23T01:19:07.6034924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1hys8a9/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6035307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6035668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6036010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6036266Z ok (2.278s)
2022-11-23T01:19:07.6036368Z 
2022-11-23T01:19:07.6036570Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6036810Z Ran 1 test in 2.279s
2022-11-23T01:19:07.6036907Z 
2022-11-23T01:19:07.6036968Z OK
2022-11-23T01:19:07.6037059Z 
2022-11-23T01:19:07.6037143Z Generating XML reports...
2022-11-23T01:19:07.6037601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010453.xml
2022-11-23T01:19:07.6038085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6038546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6038877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6039303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6039633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6039795Z 
2022-11-23T01:19:07.6039896Z Running tests...
2022-11-23T01:19:07.6040197Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6040616Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1295
2022-11-23T01:19:07.6041045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1296
2022-11-23T01:19:07.6041374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1297
2022-11-23T01:19:07.6041827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6042175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6042603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6042946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6043268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6043739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6044064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6044487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6044817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6045155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6045625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6045953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6046365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6046708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6047040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6047522Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6048042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6048556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6048943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6049280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6049624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6049979Z skip: CUDA is not available. (2.176s)
2022-11-23T01:19:07.6050111Z 
2022-11-23T01:19:07.6050313Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6050537Z Ran 1 test in 2.176s
2022-11-23T01:19:07.6050649Z 
2022-11-23T01:19:07.6050724Z OK (skipped=1)
2022-11-23T01:19:07.6050834Z 
2022-11-23T01:19:07.6050918Z Generating XML reports...
2022-11-23T01:19:07.6051364Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010457.xml
2022-11-23T01:19:07.6051861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6052320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6052684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6053100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6053444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6053607Z 
2022-11-23T01:19:07.6053680Z Running tests...
2022-11-23T01:19:07.6053968Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6054406Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1348
2022-11-23T01:19:07.6055021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1349
2022-11-23T01:19:07.6055354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1350
2022-11-23T01:19:07.6055799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6056129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6056561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6056908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6057232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6057698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6058027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6058442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6058791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6059129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6059591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6059905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6060330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6060670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6060988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6061481Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6061998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6062513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6062884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6063235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6063577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6063855Z skip: CUDA is not available. (2.179s)
2022-11-23T01:19:07.6063973Z 
2022-11-23T01:19:07.6064169Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6064408Z Ran 1 test in 2.180s
2022-11-23T01:19:07.6064520Z 
2022-11-23T01:19:07.6064592Z OK (skipped=1)
2022-11-23T01:19:07.6064700Z 
2022-11-23T01:19:07.6064770Z Generating XML reports...
2022-11-23T01:19:07.6065228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010501.xml
2022-11-23T01:19:07.6065779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6066243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6066560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6066990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6067336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6067550Z 
2022-11-23T01:19:07.6067622Z Running tests...
2022-11-23T01:19:07.6067910Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6068356Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1401
2022-11-23T01:19:07.6068804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1402
2022-11-23T01:19:07.6069127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1403
2022-11-23T01:19:07.6069577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6069905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6070336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6070667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6071007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6071478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6071792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6072219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6072565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6072900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6073354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6073683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6074110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6074453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6074775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6075272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6075792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6076295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6076680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6077034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6077384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6077648Z skip: CUDA is not available. (2.177s)
2022-11-23T01:19:07.6077781Z 
2022-11-23T01:19:07.6077981Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6078252Z Ran 1 test in 2.177s
2022-11-23T01:19:07.6078367Z 
2022-11-23T01:19:07.6078425Z OK (skipped=1)
2022-11-23T01:19:07.6078533Z 
2022-11-23T01:19:07.6078617Z Generating XML reports...
2022-11-23T01:19:07.6079075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010505.xml
2022-11-23T01:19:07.6079572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6080023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6080384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6080812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6081157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6081308Z 
2022-11-23T01:19:07.6081379Z Running tests...
2022-11-23T01:19:07.6081679Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6082139Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1454
2022-11-23T01:19:07.6082577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1455
2022-11-23T01:19:07.6082903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1456
2022-11-23T01:19:07.6083345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6083678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6084090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6084438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6084777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6085247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6085559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6085989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6086331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6086655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6087153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6087650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6087975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6088389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6088734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6089070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6089549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6090151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6090542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6090925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6091259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6091538Z skip: CUDA is not available. (2.178s)
2022-11-23T01:19:07.6091670Z 
2022-11-23T01:19:07.6091871Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6092110Z Ran 1 test in 2.179s
2022-11-23T01:19:07.6092209Z 
2022-11-23T01:19:07.6092282Z OK (skipped=1)
2022-11-23T01:19:07.6092390Z 
2022-11-23T01:19:07.6092474Z Generating XML reports...
2022-11-23T01:19:07.6092940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010509.xml
2022-11-23T01:19:07.6093462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6093920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6094255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6094685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6095200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6095365Z 
2022-11-23T01:19:07.6095439Z Running tests...
2022-11-23T01:19:07.6095745Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6096191Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1507
2022-11-23T01:19:07.6096651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1508
2022-11-23T01:19:07.6096980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1509
2022-11-23T01:19:07.6097432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6097749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6098176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6098523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6098862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6099316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6099647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6100075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6100405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6100740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6101206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6101530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6101943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6102287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6102623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6103116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6103624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6104193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6104582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6104914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6105261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6105535Z skip: CUDA is not available. (2.181s)
2022-11-23T01:19:07.6105668Z 
2022-11-23T01:19:07.6105901Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6106128Z Ran 1 test in 2.181s
2022-11-23T01:19:07.6106240Z 
2022-11-23T01:19:07.6106313Z OK (skipped=1)
2022-11-23T01:19:07.6106421Z 
2022-11-23T01:19:07.6106505Z Generating XML reports...
2022-11-23T01:19:07.6106950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010513.xml
2022-11-23T01:19:07.6107447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6107911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6108241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6108656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6109002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6109169Z 
2022-11-23T01:19:07.6109241Z Running tests...
2022-11-23T01:19:07.6109529Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6109971Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1560
2022-11-23T01:19:07.6110411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1561
2022-11-23T01:19:07.6110745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1562
2022-11-23T01:19:07.6111182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6111511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6111935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6112283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6112603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6113076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6113405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6113816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6114156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6114493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6114963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6115278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6115708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6116049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6116381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6116891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6117412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6117926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6118307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6118673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6119013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6119290Z skip: CUDA is not available. (2.181s)
2022-11-23T01:19:07.6119420Z 
2022-11-23T01:19:07.6119605Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6119847Z Ran 1 test in 2.181s
2022-11-23T01:19:07.6119959Z 
2022-11-23T01:19:07.6120031Z OK (skipped=1)
2022-11-23T01:19:07.6120138Z 
2022-11-23T01:19:07.6120223Z Generating XML reports...
2022-11-23T01:19:07.6120664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010517.xml
2022-11-23T01:19:07.6121160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6121616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6121935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6122364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6122710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6122873Z 
2022-11-23T01:19:07.6122948Z Running tests...
2022-11-23T01:19:07.6123234Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6123703Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1613
2022-11-23T01:19:07.6124158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1614
2022-11-23T01:19:07.6124475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1615
2022-11-23T01:19:07.6124932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6125264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6125695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6126030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6126366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6126833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6127164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6127577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6127924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6128260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6128713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6129043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6129498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6129924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6130245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6130745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6131264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6131818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6132189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6132542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6132884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6133148Z skip: CUDA is not available. (2.075s)
2022-11-23T01:19:07.6133279Z 
2022-11-23T01:19:07.6133477Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6133718Z Ran 1 test in 2.075s
2022-11-23T01:19:07.6133830Z 
2022-11-23T01:19:07.6133902Z OK (skipped=1)
2022-11-23T01:19:07.6133995Z 
2022-11-23T01:19:07.6134080Z Generating XML reports...
2022-11-23T01:19:07.6134540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010521.xml
2022-11-23T01:19:07.6135167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6135622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6135955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6136380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6136725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6136891Z 
2022-11-23T01:19:07.6136950Z Running tests...
2022-11-23T01:19:07.6137251Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6138160Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s)
2022-11-23T01:19:07.6138657Z 
2022-11-23T01:19:07.6138853Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6139082Z Ran 1 test in 0.261s
2022-11-23T01:19:07.6139195Z 
2022-11-23T01:19:07.6139267Z OK (skipped=1)
2022-11-23T01:19:07.6139374Z 
2022-11-23T01:19:07.6139459Z Generating XML reports...
2022-11-23T01:19:07.6139917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010525.xml
2022-11-23T01:19:07.6140399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6140859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6141193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6141611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6141958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6142122Z 
2022-11-23T01:19:07.6142245Z Running tests...
2022-11-23T01:19:07.6142547Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6142963Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1676
2022-11-23T01:19:07.6143394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1677
2022-11-23T01:19:07.6143722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1678
2022-11-23T01:19:07.6144175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6144529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6144960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6145305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6145633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6146103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6146431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6146859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6147189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6147525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6148017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6148562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6148877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6149304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6149648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6149967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6150460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6150982Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6151366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6151697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6152038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6152285Z ok (2.379s)
2022-11-23T01:19:07.6152386Z 
2022-11-23T01:19:07.6152570Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6152810Z Ran 1 test in 2.379s
2022-11-23T01:19:07.6152920Z 
2022-11-23T01:19:07.6152981Z OK
2022-11-23T01:19:07.6153075Z 
2022-11-23T01:19:07.6153161Z Generating XML reports...
2022-11-23T01:19:07.6153607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010527.xml
2022-11-23T01:19:07.6154101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6154560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6154873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6155338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6155682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6155845Z 
2022-11-23T01:19:07.6155917Z Running tests...
2022-11-23T01:19:07.6156201Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6157107Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.6157639Z 
2022-11-23T01:19:07.6157832Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6158065Z Ran 1 test in 0.254s
2022-11-23T01:19:07.6158179Z 
2022-11-23T01:19:07.6158237Z OK (skipped=1)
2022-11-23T01:19:07.6158343Z 
2022-11-23T01:19:07.6158426Z Generating XML reports...
2022-11-23T01:19:07.6158880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010531.xml
2022-11-23T01:19:07.6159373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6159822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6160152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6160580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6160912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6161076Z 
2022-11-23T01:19:07.6161147Z Running tests...
2022-11-23T01:19:07.6161449Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6161862Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1739
2022-11-23T01:19:07.6162263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1740
2022-11-23T01:19:07.6170289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1741
2022-11-23T01:19:07.6171127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6171610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6172040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6172392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6172732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6173192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6173529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6173955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6174296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6174612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6175247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6175580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6176103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6176435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6176772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6177270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6177907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6178694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6179083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6179429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6179762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6180047Z skip: Need at least 3 CUDA devices (2.077s)
2022-11-23T01:19:07.6180195Z 
2022-11-23T01:19:07.6180397Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6180639Z Ran 1 test in 2.078s
2022-11-23T01:19:07.6180739Z 
2022-11-23T01:19:07.6180811Z OK (skipped=1)
2022-11-23T01:19:07.6180917Z 
2022-11-23T01:19:07.6181001Z Generating XML reports...
2022-11-23T01:19:07.6181462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010533.xml
2022-11-23T01:19:07.6181948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6182411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6182740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6183168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6183500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6183663Z 
2022-11-23T01:19:07.6183734Z Running tests...
2022-11-23T01:19:07.6184031Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6184366Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:19:07.6184572Z 
2022-11-23T01:19:07.6184768Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6185003Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6185113Z 
2022-11-23T01:19:07.6185185Z OK (skipped=1)
2022-11-23T01:19:07.6185279Z 
2022-11-23T01:19:07.6185360Z Generating XML reports...
2022-11-23T01:19:07.6185816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010537.xml
2022-11-23T01:19:07.6186309Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6186754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6187085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6187510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6187853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6188018Z 
2022-11-23T01:19:07.6188076Z Running tests...
2022-11-23T01:19:07.6188371Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6188679Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.6189144Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1802
2022-11-23T01:19:07.6189534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1803
2022-11-23T01:19:07.6189863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1804
2022-11-23T01:19:07.6190316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6190631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6191059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6191434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6191769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6192223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6192549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6192970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6193300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6193631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6194094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6194419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6194829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6195169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6195499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6195990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6196501Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6197012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6197394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6197726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6198069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6198438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3ia3kd4
2022-11-23T01:19:07.6198830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstljwiwo
2022-11-23T01:19:07.6199197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hvgtrjf
2022-11-23T01:19:07.6199594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3ia3kd4/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6199994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstljwiwo/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6200393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hvgtrjf/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6200654Z ok (2.173s)
2022-11-23T01:19:07.6200754Z 
2022-11-23T01:19:07.6200950Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6201184Z Ran 1 test in 2.174s
2022-11-23T01:19:07.6201293Z 
2022-11-23T01:19:07.6201340Z OK
2022-11-23T01:19:07.6201430Z 
2022-11-23T01:19:07.6201514Z Generating XML reports...
2022-11-23T01:19:07.6202004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010538.xml
2022-11-23T01:19:07.6202485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6202945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6203274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6203698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6204065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6204229Z 
2022-11-23T01:19:07.6204301Z Running tests...
2022-11-23T01:19:07.6204597Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6204931Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.6205332Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1855
2022-11-23T01:19:07.6205722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1856
2022-11-23T01:19:07.6206046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1857
2022-11-23T01:19:07.6206481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6206809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6207235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6207579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6207896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6208365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6208692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6209104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6209445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6209892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6210367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6210684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6211107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6211450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6211782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6212261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6212784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6213296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6213681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6214016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6214354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6214755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rnhd_4e
2022-11-23T01:19:07.6215277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4xn62ca
2022-11-23T01:19:07.6215664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2acy5x7_
2022-11-23T01:19:07.6216058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rnhd_4e/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6216456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4xn62ca/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6216912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2acy5x7_/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6217294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6217652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6217997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6218253Z ok (2.178s)
2022-11-23T01:19:07.6218354Z 
2022-11-23T01:19:07.6218557Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6218794Z Ran 1 test in 2.179s
2022-11-23T01:19:07.6218892Z 
2022-11-23T01:19:07.6218951Z OK
2022-11-23T01:19:07.6219041Z 
2022-11-23T01:19:07.6219122Z Generating XML reports...
2022-11-23T01:19:07.6219578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010542.xml
2022-11-23T01:19:07.6220059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6220519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6220850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6221280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6221612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6221775Z 
2022-11-23T01:19:07.6221847Z Running tests...
2022-11-23T01:19:07.6222143Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6222473Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.6222908Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1908
2022-11-23T01:19:07.6223311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1909
2022-11-23T01:19:07.6223639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1910
2022-11-23T01:19:07.6224076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6224405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6224829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6225173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6225494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6225966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6226293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6226708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6227049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6227427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6227893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6228204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6228627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6228968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6229299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6229814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6230333Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6230850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6231218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6231567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6231932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxko56a7h
2022-11-23T01:19:07.6232338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxko56a7h/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6232720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q6qznn6
2022-11-23T01:19:07.6233116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q6qznn6/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6233489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6233852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmz12cpf
2022-11-23T01:19:07.6234237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmz12cpf/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6234614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6234966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6235313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.6235565Z ok (2.275s)
2022-11-23T01:19:07.6235666Z 
2022-11-23T01:19:07.6235867Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6236102Z Ran 1 test in 2.275s
2022-11-23T01:19:07.6236199Z 
2022-11-23T01:19:07.6236258Z OK
2022-11-23T01:19:07.6236348Z 
2022-11-23T01:19:07.6236429Z Generating XML reports...
2022-11-23T01:19:07.6236888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010546.xml
2022-11-23T01:19:07.6237367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6237826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6238154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6238580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6238911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6239077Z 
2022-11-23T01:19:07.6239148Z Running tests...
2022-11-23T01:19:07.6239446Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6239761Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:19:07.6240194Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1961
2022-11-23T01:19:07.6240582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1962
2022-11-23T01:19:07.6240910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1963
2022-11-23T01:19:07.6241345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6241673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6242096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6242454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6242789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6243261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6243583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6243992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6244334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6244668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6245162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6245662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6245974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6246401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6246741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6247059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6247546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6248060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6248443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6248775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6249114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6249486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmotj32g
2022-11-23T01:19:07.6249944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp230hy9n7
2022-11-23T01:19:07.6250323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmotj32g/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6250717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3pw85sf
2022-11-23T01:19:07.6251116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp230hy9n7/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6251505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3pw85sf/_remote_module_non_scriptable.py
2022-11-23T01:19:07.6251784Z ok (2.179s)
2022-11-23T01:19:07.6251790Z 
2022-11-23T01:19:07.6251992Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6252067Z Ran 1 test in 2.179s
2022-11-23T01:19:07.6252073Z 
2022-11-23T01:19:07.6252132Z OK
2022-11-23T01:19:07.6252137Z 
2022-11-23T01:19:07.6252220Z Generating XML reports...
2022-11-23T01:19:07.6252599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010550.xml
2022-11-23T01:19:07.6252822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6253098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6253219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6253501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6253666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6253672Z 
2022-11-23T01:19:07.6253744Z Running tests...
2022-11-23T01:19:07.6253940Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6254166Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2014
2022-11-23T01:19:07.6254322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2015
2022-11-23T01:19:07.6254463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2016
2022-11-23T01:19:07.6254735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6254984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6255293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6255431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6255608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6255883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6256005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6256271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6256403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6256578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6256850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6256972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6257251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6257383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6257559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6257859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6258140Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6258432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6258596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6258761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6259008Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6259302Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6259468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6259706Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6260118Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6260124Z 
2022-11-23T01:19:07.6260549Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6260590Z 
2022-11-23T01:19:07.6260823Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6261076Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6261318Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6261555Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6261790Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6262033Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6262272Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6262513Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6262762Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6263005Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6263257Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6263495Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6263890Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6263896Z 
2022-11-23T01:19:07.6264138Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6264393Z STAGE:2022-11-23 01:05:56 2014:2014 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6264636Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6264876Z STAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6265305Z STAGE:2022-11-23 01:05:56 2016:2016 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:05:56 2015:2015 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6265311Z 
2022-11-23T01:19:07.6265380Z ok (2.173s)
2022-11-23T01:19:07.6265386Z 
2022-11-23T01:19:07.6265581Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6265643Z Ran 1 test in 2.173s
2022-11-23T01:19:07.6265648Z 
2022-11-23T01:19:07.6265707Z OK
2022-11-23T01:19:07.6265714Z 
2022-11-23T01:19:07.6265796Z Generating XML reports...
2022-11-23T01:19:07.6266141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010554.xml
2022-11-23T01:19:07.6266376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6266677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6266801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6267084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6267206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6267224Z 
2022-11-23T01:19:07.6267283Z Running tests...
2022-11-23T01:19:07.6267480Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6267757Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2070
2022-11-23T01:19:07.6267913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2071
2022-11-23T01:19:07.6268063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2072
2022-11-23T01:19:07.6268338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6268460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6268740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6268862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6269038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6269310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6269433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6269712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6269849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6270025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6270294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6270403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6270680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6270812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6270988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6271285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6271584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6271875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6272038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6272200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6272433Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6272677Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6272838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6273080Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6273357Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6273599Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6273837Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6274089Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6274338Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6274606Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6274841Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6275236Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6275243Z 
2022-11-23T01:19:07.6275651Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6275895Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6275901Z 
2022-11-23T01:19:07.6276151Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6276403Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6276650Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6276885Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6277125Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6277352Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6277594Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6277836Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6278072Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6278326Z STAGE:2022-11-23 01:06:00 2070:2070 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6278579Z STAGE:2022-11-23 01:06:00 2071:2071 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6278828Z STAGE:2022-11-23 01:06:00 2072:2072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6279402Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6279554Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6280117Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6280266Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6280836Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6280984Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6281049Z ok (2.272s)
2022-11-23T01:19:07.6281055Z 
2022-11-23T01:19:07.6281253Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6281329Z Ran 1 test in 2.273s
2022-11-23T01:19:07.6281335Z 
2022-11-23T01:19:07.6281394Z OK
2022-11-23T01:19:07.6281399Z 
2022-11-23T01:19:07.6281480Z Generating XML reports...
2022-11-23T01:19:07.6281863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010558.xml
2022-11-23T01:19:07.6282095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6282358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6282481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6282762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6282896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6282901Z 
2022-11-23T01:19:07.6282975Z Running tests...
2022-11-23T01:19:07.6283169Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6283412Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2126
2022-11-23T01:19:07.6283571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2127
2022-11-23T01:19:07.6283722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2128
2022-11-23T01:19:07.6283987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6284109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6284386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6284521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6284700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6284971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6285095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6285375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6285496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6285673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6285969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6286237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6286359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6286638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6286773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6286945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6287238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6287548Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6287713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6287876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6288037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6288210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6288411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6288583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6288879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6289176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6289455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6289770Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6290177Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6290186Z 
2022-11-23T01:19:07.6290430Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6290841Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6290847Z 
2022-11-23T01:19:07.6291099Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6291353Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6291603Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6291843Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6292238Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6292244Z 
2022-11-23T01:19:07.6292645Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6292651Z 
2022-11-23T01:19:07.6292894Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6293304Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6293323Z 
2022-11-23T01:19:07.6293562Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6293802Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6294037Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6294275Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6294550Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6294804Z STAGE:2022-11-23 01:06:04 2127:2127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6295182Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6295421Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6295676Z STAGE:2022-11-23 01:06:04 2126:2126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6295966Z STAGE:2022-11-23 01:06:04 2128:2128 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6296533Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6296686Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6297248Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6297394Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6297951Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6298100Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6298170Z ok (2.373s)
2022-11-23T01:19:07.6298176Z 
2022-11-23T01:19:07.6298373Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6298448Z Ran 1 test in 2.373s
2022-11-23T01:19:07.6298454Z 
2022-11-23T01:19:07.6298500Z OK
2022-11-23T01:19:07.6298517Z 
2022-11-23T01:19:07.6298588Z Generating XML reports...
2022-11-23T01:19:07.6298927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010602.xml
2022-11-23T01:19:07.6299161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6299437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6299560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6299844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6299980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6299986Z 
2022-11-23T01:19:07.6300057Z Running tests...
2022-11-23T01:19:07.6300238Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6300479Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2191
2022-11-23T01:19:07.6300634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2192
2022-11-23T01:19:07.6300791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2193
2022-11-23T01:19:07.6301072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6301195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6301513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6301649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6301826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6302087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6302209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6302489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6302653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6302826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6303098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6303221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6303499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6303621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6303796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6304094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6304393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6304682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6304849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6305015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6305177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6305349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6305509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6305677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6305979Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6306279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6306526Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6306817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6307055Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6307466Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6307472Z 
2022-11-23T01:19:07.6307903Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6307908Z 
2022-11-23T01:19:07.6308148Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6308416Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6308649Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6308888Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6309142Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6309396Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6309663Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6309901Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6310146Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6310390Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6310643Z STAGE:2022-11-23 01:06:08 2192:2192 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6310879Z STAGE:2022-11-23 01:06:08 2193:2193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6311440Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6311595Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6312148Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6312297Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6312366Z ok (2.271s)
2022-11-23T01:19:07.6312371Z 
2022-11-23T01:19:07.6312566Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6312640Z Ran 1 test in 2.271s
2022-11-23T01:19:07.6312646Z 
2022-11-23T01:19:07.6312705Z OK
2022-11-23T01:19:07.6312711Z 
2022-11-23T01:19:07.6312793Z Generating XML reports...
2022-11-23T01:19:07.6313120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010606.xml
2022-11-23T01:19:07.6313357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6313633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6313758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6314040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6314173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6314179Z 
2022-11-23T01:19:07.6314250Z Running tests...
2022-11-23T01:19:07.6314444Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6314674Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2252
2022-11-23T01:19:07.6314831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2253
2022-11-23T01:19:07.6314985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2254
2022-11-23T01:19:07.6315256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6315407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6315687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6315823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6316000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6316271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6316408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6316689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6316824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6317006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6317303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6317570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6317691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6317972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6318105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6318268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6318562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6318860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6319031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6319191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6319435Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6319594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6319835Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6320073Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6320304Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6320544Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6320781Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6321036Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6321286Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6321536Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6321774Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6322006Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6322231Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6322500Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6322902Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6322909Z 
2022-11-23T01:19:07.6323159Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6323406Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6323718Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6323957Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6324192Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6324431Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6324672Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6324899Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6325136Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6325388Z STAGE:2022-11-23 01:06:12 2254:2254 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6325642Z STAGE:2022-11-23 01:06:12 2252:2252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6325891Z STAGE:2022-11-23 01:06:12 2253:2253 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6326458Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6326608Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6327173Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6327321Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6327869Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6328019Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6328075Z ok (2.175s)
2022-11-23T01:19:07.6328080Z 
2022-11-23T01:19:07.6328276Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6328350Z Ran 1 test in 2.175s
2022-11-23T01:19:07.6328355Z 
2022-11-23T01:19:07.6328415Z OK
2022-11-23T01:19:07.6328420Z 
2022-11-23T01:19:07.6328501Z Generating XML reports...
2022-11-23T01:19:07.6328845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010610.xml
2022-11-23T01:19:07.6329079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6329354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6329464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6329872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6330012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6330017Z 
2022-11-23T01:19:07.6330088Z Running tests...
2022-11-23T01:19:07.6330288Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6330536Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2308
2022-11-23T01:19:07.6330733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2309
2022-11-23T01:19:07.6330885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2310
2022-11-23T01:19:07.6331160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6331270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6331549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6331683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6331858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6332129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6332251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6332533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6332665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6332839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6333098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6333218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6333497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6333630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6333803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6334104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6334402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6334696Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6334973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6335141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6335304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6335553Z STAGE:2022-11-23 01:06:16 2308:2308 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6335795Z STAGE:2022-11-23 01:06:16 2309:2309 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6336037Z STAGE:2022-11-23 01:06:16 2310:2310 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6336280Z STAGE:2022-11-23 01:06:16 2310:2310 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6336522Z STAGE:2022-11-23 01:06:16 2309:2309 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6336825Z STAGE:2022-11-23 01:06:16 2310:2310 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6337057Z STAGE:2022-11-23 01:06:16 2308:2308 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6337307Z STAGE:2022-11-23 01:06:16 2309:2309 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6337562Z STAGE:2022-11-23 01:06:16 2308:2308 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6338129Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6338315Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6338873Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6339022Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6339581Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6339727Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6339793Z ok (2.174s)
2022-11-23T01:19:07.6339798Z 
2022-11-23T01:19:07.6339988Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6340051Z Ran 1 test in 2.175s
2022-11-23T01:19:07.6340057Z 
2022-11-23T01:19:07.6340117Z OK
2022-11-23T01:19:07.6340122Z 
2022-11-23T01:19:07.6340205Z Generating XML reports...
2022-11-23T01:19:07.6340549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010614.xml
2022-11-23T01:19:07.6340780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6341053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6341176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6341460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6341593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6341599Z 
2022-11-23T01:19:07.6341658Z Running tests...
2022-11-23T01:19:07.6341855Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6342089Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2364
2022-11-23T01:19:07.6342245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2365
2022-11-23T01:19:07.6342396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2366
2022-11-23T01:19:07.6342667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6342788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6343070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6343191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6343392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6343665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6343788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6344069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6344201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6344374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6344700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6344970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6345079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6345356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6345490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6345662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6345959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6346254Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6346423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6346583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6346743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6346975Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6347215Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6347456Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6347700Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6347957Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6348203Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6348443Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6348868Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6348874Z 
2022-11-23T01:19:07.6349114Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6349496Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6349514Z 
2022-11-23T01:19:07.6349741Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6349995Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6350237Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6350504Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6350756Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6351005Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6351242Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6351632Z STAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6351665Z 
2022-11-23T01:19:07.6351907Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6352158Z STAGE:2022-11-23 01:06:20 2364:2364 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6352551Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6352569Z 
2022-11-23T01:19:07.6352978Z STAGE:2022-11-23 01:06:20 2366:2366 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:20 2365:2365 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6352996Z 
2022-11-23T01:19:07.6353049Z ok (2.176s)
2022-11-23T01:19:07.6353055Z 
2022-11-23T01:19:07.6353250Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6353326Z Ran 1 test in 2.176s
2022-11-23T01:19:07.6353332Z 
2022-11-23T01:19:07.6353390Z OK
2022-11-23T01:19:07.6353395Z 
2022-11-23T01:19:07.6353476Z Generating XML reports...
2022-11-23T01:19:07.6353816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010618.xml
2022-11-23T01:19:07.6354047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6354324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6354435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6354716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6354849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6354856Z 
2022-11-23T01:19:07.6354928Z Running tests...
2022-11-23T01:19:07.6355122Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6355315Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-11-23T01:19:07.6355321Z 
2022-11-23T01:19:07.6355513Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6355587Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6355592Z 
2022-11-23T01:19:07.6355650Z OK (skipped=1)
2022-11-23T01:19:07.6355667Z 
2022-11-23T01:19:07.6355738Z Generating XML reports...
2022-11-23T01:19:07.6356077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010622.xml
2022-11-23T01:19:07.6356306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6356584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6356708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6356985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6357118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6357124Z 
2022-11-23T01:19:07.6357194Z Running tests...
2022-11-23T01:19:07.6357404Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6357602Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-11-23T01:19:07.6357607Z 
2022-11-23T01:19:07.6357797Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6357872Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6357877Z 
2022-11-23T01:19:07.6357948Z OK (skipped=1)
2022-11-23T01:19:07.6357953Z 
2022-11-23T01:19:07.6358034Z Generating XML reports...
2022-11-23T01:19:07.6358397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010623.xml
2022-11-23T01:19:07.6358625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6358900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6359013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6359292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6359425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6359431Z 
2022-11-23T01:19:07.6359501Z Running tests...
2022-11-23T01:19:07.6359691Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6359923Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2440
2022-11-23T01:19:07.6360079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2441
2022-11-23T01:19:07.6360232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2442
2022-11-23T01:19:07.6360505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6360617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6360899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6361031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6361211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6361485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6361607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6361888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6362020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6362186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6362484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6362755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6362877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6363156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6363290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6363463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6363762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6364077Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6364231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6364393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6364555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6364727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6364926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6365223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6365397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6365695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6365984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6366214Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6366455Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6366696Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6367107Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6367114Z 
2022-11-23T01:19:07.6367359Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6367780Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6367785Z 
2022-11-23T01:19:07.6368037Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6368277Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6368513Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6368749Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6368975Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6369212Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6369449Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6369763Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6370021Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6370267Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6370662Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6370668Z 
2022-11-23T01:19:07.6370903Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6371180Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6371420Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6371648Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6371901Z STAGE:2022-11-23 01:06:27 2442:2442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6372147Z STAGE:2022-11-23 01:06:27 2441:2441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6372435Z STAGE:2022-11-23 01:06:27 2440:2440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6372501Z ok (2.272s)
2022-11-23T01:19:07.6372506Z 
2022-11-23T01:19:07.6372700Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6372775Z Ran 1 test in 2.273s
2022-11-23T01:19:07.6372780Z 
2022-11-23T01:19:07.6372838Z OK
2022-11-23T01:19:07.6372845Z 
2022-11-23T01:19:07.6372927Z Generating XML reports...
2022-11-23T01:19:07.6373257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010625.xml
2022-11-23T01:19:07.6373487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6373762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6373883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6374168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6374301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6374306Z 
2022-11-23T01:19:07.6374378Z Running tests...
2022-11-23T01:19:07.6374572Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6374792Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2505
2022-11-23T01:19:07.6375087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2506
2022-11-23T01:19:07.6375245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2507
2022-11-23T01:19:07.6375520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6375644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6375929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6376063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6376239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6376512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6376622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6376901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6377032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6377204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6377477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6377598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6377876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6378058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6378220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6378518Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6405575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6406001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6406306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6406466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6406635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6406804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6406955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6407116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6407415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6407706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6407998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6408236Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6408476Z STAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6408716Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6408949Z STAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6409193Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6409437Z STAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6409668Z STAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6409988Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6410391Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6410399Z 
2022-11-23T01:19:07.6410645Z STAGE:2022-11-23 01:06:31 2507:2507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6410887Z STAGE:2022-11-23 01:06:31 2506:2506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6410946Z ok (2.276s)
2022-11-23T01:19:07.6410952Z 
2022-11-23T01:19:07.6411139Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6411202Z Ran 1 test in 2.276s
2022-11-23T01:19:07.6411208Z 
2022-11-23T01:19:07.6411267Z OK
2022-11-23T01:19:07.6411272Z 
2022-11-23T01:19:07.6411349Z Generating XML reports...
2022-11-23T01:19:07.6411686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010629.xml
2022-11-23T01:19:07.6411911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6412228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6412346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6412622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6412748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6412754Z 
2022-11-23T01:19:07.6412815Z Running tests...
2022-11-23T01:19:07.6413003Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6413241Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:19:07.6413247Z 
2022-11-23T01:19:07.6413432Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6413498Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6413503Z 
2022-11-23T01:19:07.6413568Z OK (skipped=1)
2022-11-23T01:19:07.6413574Z 
2022-11-23T01:19:07.6413649Z Generating XML reports...
2022-11-23T01:19:07.6413983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010633.xml
2022-11-23T01:19:07.6414201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6414468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6414582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6414974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6415131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6415137Z 
2022-11-23T01:19:07.6415202Z Running tests...
2022-11-23T01:19:07.6415397Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6415606Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:19:07.6415612Z 
2022-11-23T01:19:07.6415794Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6415856Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6415866Z 
2022-11-23T01:19:07.6415922Z OK (skipped=1)
2022-11-23T01:19:07.6415927Z 
2022-11-23T01:19:07.6416001Z Generating XML reports...
2022-11-23T01:19:07.6416334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010635.xml
2022-11-23T01:19:07.6416558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6416826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6416944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6417215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6417341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6417347Z 
2022-11-23T01:19:07.6417406Z Running tests...
2022-11-23T01:19:07.6417592Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6417791Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:19:07.6417799Z 
2022-11-23T01:19:07.6417980Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6418045Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6418051Z 
2022-11-23T01:19:07.6418114Z OK (skipped=1)
2022-11-23T01:19:07.6418118Z 
2022-11-23T01:19:07.6418191Z Generating XML reports...
2022-11-23T01:19:07.6418570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010636.xml
2022-11-23T01:19:07.6418796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6419056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6419171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6419442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6419602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6419608Z 
2022-11-23T01:19:07.6419670Z Running tests...
2022-11-23T01:19:07.6419856Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6420065Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:19:07.6420074Z 
2022-11-23T01:19:07.6420258Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6420323Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6420328Z 
2022-11-23T01:19:07.6420384Z OK (skipped=1)
2022-11-23T01:19:07.6420389Z 
2022-11-23T01:19:07.6420462Z Generating XML reports...
2022-11-23T01:19:07.6420795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010638.xml
2022-11-23T01:19:07.6421017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6421284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6421398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6421670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6421798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6421804Z 
2022-11-23T01:19:07.6421864Z Running tests...
2022-11-23T01:19:07.6422051Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6422286Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2606
2022-11-23T01:19:07.6422434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2607
2022-11-23T01:19:07.6422576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2608
2022-11-23T01:19:07.6422845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6422959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6423234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6423359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6423524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6423788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6423902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6424182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6424317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6424496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6424768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6424915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6425195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6425316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6425490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6425790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6426113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6426406Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6426569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6426735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6426893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6426946Z ok (2.175s)
2022-11-23T01:19:07.6426963Z 
2022-11-23T01:19:07.6427145Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6427219Z Ran 1 test in 2.175s
2022-11-23T01:19:07.6427224Z 
2022-11-23T01:19:07.6427283Z OK
2022-11-23T01:19:07.6427288Z 
2022-11-23T01:19:07.6427371Z Generating XML reports...
2022-11-23T01:19:07.6427716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010639.xml
2022-11-23T01:19:07.6427948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6428223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6428345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6428614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6428747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6428752Z 
2022-11-23T01:19:07.6428820Z Running tests...
2022-11-23T01:19:07.6429012Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6429250Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2659
2022-11-23T01:19:07.6429406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2660
2022-11-23T01:19:07.6429558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2661
2022-11-23T01:19:07.6429833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6429954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6430223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6430357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6430533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6430807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6430929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6431209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6431343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6431547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6431832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6432105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6432226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6432506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6432664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6432840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6433137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6433433Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6433598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6433746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6433909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6434082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6434260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6434437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6434733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6435026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6435314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6435483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:19:07.6435641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:19:07.6435940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:19:07.6436109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:19:07.6436401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:19:07.6436699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:19:07.6436871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:19:07.6437044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:19:07.6437213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:19:07.6437502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:19:07.6437796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:19:07.6438079Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:19:07.6438174Z ok (2.278s)
2022-11-23T01:19:07.6438180Z 
2022-11-23T01:19:07.6438377Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6438451Z Ran 1 test in 2.278s
2022-11-23T01:19:07.6438457Z 
2022-11-23T01:19:07.6438516Z OK
2022-11-23T01:19:07.6438521Z 
2022-11-23T01:19:07.6438604Z Generating XML reports...
2022-11-23T01:19:07.6438945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010643.xml
2022-11-23T01:19:07.6439181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6439469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6439592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6439871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6440008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6440014Z 
2022-11-23T01:19:07.6440085Z Running tests...
2022-11-23T01:19:07.6440279Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6440469Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-11-23T01:19:07.6440475Z 
2022-11-23T01:19:07.6440664Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6440738Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6440746Z 
2022-11-23T01:19:07.6440803Z OK (skipped=1)
2022-11-23T01:19:07.6440808Z 
2022-11-23T01:19:07.6440891Z Generating XML reports...
2022-11-23T01:19:07.6441230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010647.xml
2022-11-23T01:19:07.6441463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6441738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6441860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6442140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6442273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6442279Z 
2022-11-23T01:19:07.6442351Z Running tests...
2022-11-23T01:19:07.6442531Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6442784Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2758
2022-11-23T01:19:07.6442941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2759
2022-11-23T01:19:07.6443095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2760
2022-11-23T01:19:07.6443369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6443491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6443772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6443906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6444070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6444344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6444467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6444771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6444907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6445082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6445381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6445652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6445772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6446064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6446197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6446370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6446670Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6446964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6447128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6447284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6447438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6447614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6447774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6447949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6448240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6448536Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6448827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6449073Z STAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6449319Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6449560Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6450047Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6450281Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6450301Z 
2022-11-23T01:19:07.6450882Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6450901Z 
2022-11-23T01:19:07.6450908Z 
2022-11-23T01:19:07.6451132Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6451369Z STAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6451605Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6452045Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6452052Z 
2022-11-23T01:19:07.6452296Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6452551Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6452804Z STAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6453095Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6453487Z STAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6453493Z 
2022-11-23T01:19:07.6453734Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6454138Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6454144Z 
2022-11-23T01:19:07.6454385Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6454797Z STAGE:2022-11-23 01:06:51 2759:2759 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:51 2760:2760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6454818Z 
2022-11-23T01:19:07.6455194Z STAGE:2022-11-23 01:06:51 2758:2758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6455773Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6455927Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6456478Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6456628Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6457194Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6457340Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6457410Z ok (2.277s)
2022-11-23T01:19:07.6457415Z 
2022-11-23T01:19:07.6457613Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6457688Z Ran 1 test in 2.278s
2022-11-23T01:19:07.6457694Z 
2022-11-23T01:19:07.6457741Z OK
2022-11-23T01:19:07.6457746Z 
2022-11-23T01:19:07.6457830Z Generating XML reports...
2022-11-23T01:19:07.6458174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010649.xml
2022-11-23T01:19:07.6458409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6458684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6458808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6459171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6459308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6459314Z 
2022-11-23T01:19:07.6459384Z Running tests...
2022-11-23T01:19:07.6459570Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6459818Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2823
2022-11-23T01:19:07.6460009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2824
2022-11-23T01:19:07.6460163Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2825
2022-11-23T01:19:07.6460439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6460565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6460843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6460978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6461142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6461416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6461538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6461819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6461952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6462129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6462407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6462529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6462810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6462930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6463104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6463407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6463705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6464003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6464168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6464334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6464497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6464670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6464832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6465007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6465304Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6465640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6465935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6466179Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6466580Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6466620Z 
2022-11-23T01:19:07.6466868Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6467113Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6467352Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6467595Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6467848Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6468099Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6468336Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6468573Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6468812Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6469221Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6469227Z 
2022-11-23T01:19:07.6469471Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6469724Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6469962Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6470212Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6470448Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6470687Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6470925Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6471172Z STAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6471416Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6471827Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:55 2823:2823 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6471833Z 
2022-11-23T01:19:07.6472086Z STAGE:2022-11-23 01:06:55 2825:2825 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6472337Z STAGE:2022-11-23 01:06:55 2824:2824 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6472894Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6473076Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6473634Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6473780Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6474338Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6474521Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6474588Z ok (2.270s)
2022-11-23T01:19:07.6474594Z 
2022-11-23T01:19:07.6474792Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6474866Z Ran 1 test in 2.271s
2022-11-23T01:19:07.6474872Z 
2022-11-23T01:19:07.6474930Z OK
2022-11-23T01:19:07.6474935Z 
2022-11-23T01:19:07.6475017Z Generating XML reports...
2022-11-23T01:19:07.6475347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010653.xml
2022-11-23T01:19:07.6475581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6475855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6475981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6476264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6476398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6476406Z 
2022-11-23T01:19:07.6476478Z Running tests...
2022-11-23T01:19:07.6476671Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6476910Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2888
2022-11-23T01:19:07.6477066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2889
2022-11-23T01:19:07.6477218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2890
2022-11-23T01:19:07.6477495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6477617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6477898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6478035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6478212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6478481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6478590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6478869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6479003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6479181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6479483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6479783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6479904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6480181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6480316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6480477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6480776Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6481103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6481266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6481431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6481591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6481763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6481938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6482115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6482397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6482697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6482992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6483241Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6483484Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6483724Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6483969Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6484378Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6484386Z 
2022-11-23T01:19:07.6484639Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6485066Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6485072Z 
2022-11-23T01:19:07.6485296Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6485534Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6485773Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6486014Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6486253Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6486505Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6486777Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6487029Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6487278Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6487503Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6487740Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6487975Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6488244Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6488486Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6488728Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6488978Z STAGE:2022-11-23 01:06:59 2890:2890 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6489229Z STAGE:2022-11-23 01:06:59 2889:2889 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6489479Z STAGE:2022-11-23 01:06:59 2888:2888 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6490105Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6490260Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6490820Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6490966Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6491521Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6491671Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6491741Z ok (2.273s)
2022-11-23T01:19:07.6491747Z 
2022-11-23T01:19:07.6491944Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6492020Z Ran 1 test in 2.274s
2022-11-23T01:19:07.6492026Z 
2022-11-23T01:19:07.6492086Z OK
2022-11-23T01:19:07.6492091Z 
2022-11-23T01:19:07.6492174Z Generating XML reports...
2022-11-23T01:19:07.6492505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010657.xml
2022-11-23T01:19:07.6492735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6493008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6493133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6493414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6493551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6493556Z 
2022-11-23T01:19:07.6493629Z Running tests...
2022-11-23T01:19:07.6493825Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6494091Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2953
2022-11-23T01:19:07.6494250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2954
2022-11-23T01:19:07.6494401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2955
2022-11-23T01:19:07.6494675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6494798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6495210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6495397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6495574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6495850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6495960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6496238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6496372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6496548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6496849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6497122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6497244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6497529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6497662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6497821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6498117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6498415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6498581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6498745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6498905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6499082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6499253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6499411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6499702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6499995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6500285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6500688Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6500694Z 
2022-11-23T01:19:07.6500969Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6501216Z STAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6501623Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6501628Z 
2022-11-23T01:19:07.6501881Z STAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6502179Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6502431Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6502659Z STAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6502896Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6503134Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6503377Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6503786Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6503794Z 
2022-11-23T01:19:07.6504046Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6504295Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6504547Z STAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6504939Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6504944Z 
2022-11-23T01:19:07.6505179Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6505421Z STAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6505648Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6506060Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:03 2954:2954 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6506066Z 
2022-11-23T01:19:07.6506318Z STAGE:2022-11-23 01:07:03 2955:2955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6506567Z STAGE:2022-11-23 01:07:03 2953:2953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6507134Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6507286Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6507843Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6507990Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6508574Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6508719Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6508786Z ok (2.179s)
2022-11-23T01:19:07.6508791Z 
2022-11-23T01:19:07.6508975Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6509079Z Ran 1 test in 2.179s
2022-11-23T01:19:07.6509085Z 
2022-11-23T01:19:07.6509145Z OK
2022-11-23T01:19:07.6509150Z 
2022-11-23T01:19:07.6509232Z Generating XML reports...
2022-11-23T01:19:07.6509572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010701.xml
2022-11-23T01:19:07.6509806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6510079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6510203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6510485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6510609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6510615Z 
2022-11-23T01:19:07.6510686Z Running tests...
2022-11-23T01:19:07.6510882Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6511129Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3018
2022-11-23T01:19:07.6511285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3019
2022-11-23T01:19:07.6511440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3020
2022-11-23T01:19:07.6511714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6511838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6512107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6512243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6512421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6512696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6512817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6513102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6513237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6513415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6513686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6513794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6514076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6514213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6514390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6514716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6515017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6515310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6515475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6515638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6515826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6516001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6516161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6516336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6516630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6516923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6517212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6517456Z STAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6517701Z STAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6518096Z STAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6518116Z 
2022-11-23T01:19:07.6518534Z STAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6518555Z 
2022-11-23T01:19:07.6518780Z STAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6519018Z STAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6519263Z STAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6519515Z STAGE:2022-11-23 01:07:07 3019:3019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6519756Z STAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6520009Z STAGE:2022-11-23 01:07:07 3020:3020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6520575Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6520727Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6521275Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6521424Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6521478Z ok (2.381s)
2022-11-23T01:19:07.6521483Z 
2022-11-23T01:19:07.6521710Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6521786Z Ran 1 test in 2.382s
2022-11-23T01:19:07.6521792Z 
2022-11-23T01:19:07.6521851Z OK
2022-11-23T01:19:07.6521857Z 
2022-11-23T01:19:07.6521939Z Generating XML reports...
2022-11-23T01:19:07.6522280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010705.xml
2022-11-23T01:19:07.6522515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6522790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6522931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6523215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6523349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6523355Z 
2022-11-23T01:19:07.6523430Z Running tests...
2022-11-23T01:19:07.6523624Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6523870Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3079
2022-11-23T01:19:07.6524028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3080
2022-11-23T01:19:07.6524180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3081
2022-11-23T01:19:07.6524454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6524566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6524846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6524981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6525161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6525435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6525556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6525836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6525968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6526147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6526404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6526525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6526805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6526938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6527111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6527407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6527702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6527994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6528158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6528308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6528508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6528682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6528842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6529015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6529305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6529626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6529987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6530239Z STAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6530470Z STAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6530720Z STAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6530961Z STAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6531216Z STAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6531472Z STAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6531869Z STAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6531875Z 
2022-11-23T01:19:07.6532282Z STAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6532287Z 
2022-11-23T01:19:07.6532537Z STAGE:2022-11-23 01:07:11 3081:3081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6532790Z STAGE:2022-11-23 01:07:11 3080:3080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6533354Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6533508Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6534054Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6534204Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6534271Z ok (2.378s)
2022-11-23T01:19:07.6534276Z 
2022-11-23T01:19:07.6534472Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6534547Z Ran 1 test in 2.379s
2022-11-23T01:19:07.6534552Z 
2022-11-23T01:19:07.6534612Z OK
2022-11-23T01:19:07.6534620Z 
2022-11-23T01:19:07.6534704Z Generating XML reports...
2022-11-23T01:19:07.6535228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010709.xml
2022-11-23T01:19:07.6535509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6535830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6535957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6536244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6536379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6536386Z 
2022-11-23T01:19:07.6536457Z Running tests...
2022-11-23T01:19:07.6536655Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6536949Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3140
2022-11-23T01:19:07.6537106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3141
2022-11-23T01:19:07.6537258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3142
2022-11-23T01:19:07.6537522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6537644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6537924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6538057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6538233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6538504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6538627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6538909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6539032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6539208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6539477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6539599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6539877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6540010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6540188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6540486Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6540785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6541062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6541230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6541388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6541560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6541739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6541901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6542071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6542393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6542689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6542966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6543211Z STAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6543456Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6543731Z STAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6543974Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6544230Z STAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6544482Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6544718Z STAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6544956Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6545350Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6545373Z 
2022-11-23T01:19:07.6545783Z STAGE:2022-11-23 01:07:15 3142:3142 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:15 3141:3141 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6545803Z 
2022-11-23T01:19:07.6546356Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6546510Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6547069Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6547221Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6547289Z ok (2.277s)
2022-11-23T01:19:07.6547294Z 
2022-11-23T01:19:07.6547491Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6547565Z Ran 1 test in 2.277s
2022-11-23T01:19:07.6547570Z 
2022-11-23T01:19:07.6547631Z OK
2022-11-23T01:19:07.6547636Z 
2022-11-23T01:19:07.6547718Z Generating XML reports...
2022-11-23T01:19:07.6548049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010713.xml
2022-11-23T01:19:07.6548283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6548557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6548681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6548965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6549098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6549104Z 
2022-11-23T01:19:07.6549176Z Running tests...
2022-11-23T01:19:07.6549398Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6549644Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3201
2022-11-23T01:19:07.6549787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3202
2022-11-23T01:19:07.6549941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3203
2022-11-23T01:19:07.6550215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6550337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6550660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6550794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6550970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6551247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6551369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6551635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6551768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6551945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6552246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6552519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6552642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6552923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6553054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6553215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6553508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6553805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6553972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6554137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6554310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6554483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6554645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6554821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6555105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6555401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6555695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6555939Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6556210Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6556458Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6556699Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6556955Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6557204Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6557467Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6557694Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6557938Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6558178Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6558433Z STAGE:2022-11-23 01:07:19 3203:3203 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6558686Z STAGE:2022-11-23 01:07:19 3202:3202 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6559252Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6559406Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6559964Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6560112Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6560179Z ok (2.375s)
2022-11-23T01:19:07.6560184Z 
2022-11-23T01:19:07.6560368Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6560442Z Ran 1 test in 2.375s
2022-11-23T01:19:07.6560447Z 
2022-11-23T01:19:07.6560506Z OK
2022-11-23T01:19:07.6560511Z 
2022-11-23T01:19:07.6560594Z Generating XML reports...
2022-11-23T01:19:07.6560937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010717.xml
2022-11-23T01:19:07.6561171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6561445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6561570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6561838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6561973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6561979Z 
2022-11-23T01:19:07.6562050Z Running tests...
2022-11-23T01:19:07.6562244Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6562484Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3262
2022-11-23T01:19:07.6562641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3263
2022-11-23T01:19:07.6562793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3264
2022-11-23T01:19:07.6563064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6563213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6563482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6563616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6563794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6564065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6564250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6564529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6564663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6564841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6565097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6565220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6565498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6565632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6565806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6566107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6566405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6566697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6566862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6567012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6567175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6567421Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6567664Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6567903Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6568147Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6568391Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6568628Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6568882Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6569118Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6569370Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6569836Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6569842Z 
2022-11-23T01:19:07.6570086Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6570366Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6570608Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6570848Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6571102Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6571352Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6571628Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6571852Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6572090Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6572329Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6572572Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6572815Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6573055Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6573306Z STAGE:2022-11-23 01:07:23 3263:3263 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6573562Z STAGE:2022-11-23 01:07:23 3262:3262 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6573815Z STAGE:2022-11-23 01:07:23 3264:3264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6574375Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6574527Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6575230Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6575382Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6575948Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6576097Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6576166Z ok (2.172s)
2022-11-23T01:19:07.6576172Z 
2022-11-23T01:19:07.6576370Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6576445Z Ran 1 test in 2.173s
2022-11-23T01:19:07.6576450Z 
2022-11-23T01:19:07.6576509Z OK
2022-11-23T01:19:07.6576515Z 
2022-11-23T01:19:07.6576584Z Generating XML reports...
2022-11-23T01:19:07.6576929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010721.xml
2022-11-23T01:19:07.6577164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6577442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6577568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6577908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6578046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6578052Z 
2022-11-23T01:19:07.6578124Z Running tests...
2022-11-23T01:19:07.6578323Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6578571Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3318
2022-11-23T01:19:07.6578763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3319
2022-11-23T01:19:07.6578916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3320
2022-11-23T01:19:07.6579195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6579321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6579604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6579740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6579921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6580180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6580304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6580588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6580721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6580897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6581170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6581294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6581573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6581709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6581873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6582175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6582470Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6582762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6582929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6583096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6583259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6583824Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6583975Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6584555Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6584705Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6585255Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6585401Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6585505Z ok (2.179s)
2022-11-23T01:19:07.6585511Z 
2022-11-23T01:19:07.6585708Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6585783Z Ran 1 test in 2.179s
2022-11-23T01:19:07.6585788Z 
2022-11-23T01:19:07.6585849Z OK
2022-11-23T01:19:07.6585853Z 
2022-11-23T01:19:07.6585936Z Generating XML reports...
2022-11-23T01:19:07.6586281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010725.xml
2022-11-23T01:19:07.6586512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6586773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6586897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6587176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6587313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6587319Z 
2022-11-23T01:19:07.6587391Z Running tests...
2022-11-23T01:19:07.6587587Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6587827Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3371
2022-11-23T01:19:07.6587984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3372
2022-11-23T01:19:07.6588123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3373
2022-11-23T01:19:07.6588399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6588519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6588800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6588936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6589112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6589382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6589504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6589785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6589905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6590082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6590352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6590477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6590756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6590888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6591095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6591393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6591691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6591973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6592139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6592414Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6592573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6592734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6592976Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6593218Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6593462Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6593688Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6593943Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6594359Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6594366Z 
2022-11-23T01:19:07.6594619Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6594860Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6595098Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6595333Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6595576Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6595830Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6596073Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6596297Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6596549Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6596800Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6597039Z STAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6597433Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6597439Z 
2022-11-23T01:19:07.6597843Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6597851Z 
2022-11-23T01:19:07.6598301Z STAGE:2022-11-23 01:07:31 3371:3371 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:31 3372:3372 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6598307Z 
2022-11-23T01:19:07.6598552Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6598806Z STAGE:2022-11-23 01:07:31 3373:3373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6599371Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6599565Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6600115Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6600263Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6600805Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6600949Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6601016Z ok (2.372s)
2022-11-23T01:19:07.6601024Z 
2022-11-23T01:19:07.6601220Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6601295Z Ran 1 test in 2.372s
2022-11-23T01:19:07.6601300Z 
2022-11-23T01:19:07.6601360Z OK
2022-11-23T01:19:07.6601365Z 
2022-11-23T01:19:07.6601446Z Generating XML reports...
2022-11-23T01:19:07.6601789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010729.xml
2022-11-23T01:19:07.6602024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6602286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6602409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6602691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6602826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6602834Z 
2022-11-23T01:19:07.6602905Z Running tests...
2022-11-23T01:19:07.6603097Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6603341Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3427
2022-11-23T01:19:07.6603498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3428
2022-11-23T01:19:07.6603651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3429
2022-11-23T01:19:07.6603910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6604033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6604317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6604455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6604633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6604906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6605057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6605337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6605458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6605635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6605907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6606027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6606341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6606475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6606652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6606957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6607256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6607539Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6607708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6607873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6608035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6608280Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6608524Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6608765Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6609009Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6609263Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6609494Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6609809Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6610070Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6610324Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6610722Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6610728Z 
2022-11-23T01:19:07.6610967Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6611214Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6611467Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6611876Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6611882Z 
2022-11-23T01:19:07.6612339Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6612345Z 
2022-11-23T01:19:07.6612582Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6612959Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6612978Z 
2022-11-23T01:19:07.6613207Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6613488Z STAGE:2022-11-23 01:07:35 3427:3427 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6613897Z STAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6613902Z 
2022-11-23T01:19:07.6614328Z STAGE:2022-11-23 01:07:35 3428:3428 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:35 3429:3429 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6614333Z 
2022-11-23T01:19:07.6615034Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6615190Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6615759Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6615910Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6616465Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6616609Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6616678Z ok (2.275s)
2022-11-23T01:19:07.6616683Z 
2022-11-23T01:19:07.6616880Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6616943Z Ran 1 test in 2.275s
2022-11-23T01:19:07.6616948Z 
2022-11-23T01:19:07.6617010Z OK
2022-11-23T01:19:07.6617015Z 
2022-11-23T01:19:07.6617099Z Generating XML reports...
2022-11-23T01:19:07.6617443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010733.xml
2022-11-23T01:19:07.6617679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6617955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6618081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6618363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6618497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6618505Z 
2022-11-23T01:19:07.6618564Z Running tests...
2022-11-23T01:19:07.6618757Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6618999Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3483
2022-11-23T01:19:07.6619204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3484
2022-11-23T01:19:07.6619361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3485
2022-11-23T01:19:07.6619635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6619761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6620045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6620168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6620380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6620679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6620955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6621077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6621360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6621494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6621669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6621944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6622055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6622336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6622470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6622647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6622946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6623244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6623408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6623656Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6623819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6624047Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6624203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6624446Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6624858Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6624864Z 
2022-11-23T01:19:07.6625110Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6625364Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6625614Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6625866Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6626108Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6626357Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6626594Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6626835Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6627410Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6627443Z 
2022-11-23T01:19:07.6627447Z 
2022-11-23T01:19:07.6627702Z STAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6627954Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6628193Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6628584Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6628589Z 
2022-11-23T01:19:07.6628994Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6629002Z 
2022-11-23T01:19:07.6629248Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6629672Z STAGE:2022-11-23 01:07:39 3485:3485 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:39 3484:3484 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6629680Z 
2022-11-23T01:19:07.6629931Z STAGE:2022-11-23 01:07:39 3483:3483 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6630496Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6630650Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6631190Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6631342Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6631900Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:19:07.6632046Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:19:07.6632113Z ok (2.272s)
2022-11-23T01:19:07.6632118Z 
2022-11-23T01:19:07.6632321Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6632396Z Ran 1 test in 2.272s
2022-11-23T01:19:07.6632403Z 
2022-11-23T01:19:07.6632461Z OK
2022-11-23T01:19:07.6632466Z 
2022-11-23T01:19:07.6632549Z Generating XML reports...
2022-11-23T01:19:07.6632890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010737.xml
2022-11-23T01:19:07.6633109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6633416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6633541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6633822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6633960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6633966Z 
2022-11-23T01:19:07.6634036Z Running tests...
2022-11-23T01:19:07.6634234Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6634512Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3539
2022-11-23T01:19:07.6634667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3540
2022-11-23T01:19:07.6634809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3541
2022-11-23T01:19:07.6635085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6635209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6635491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6635625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6635802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6636077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6636199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6636468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6636600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6636779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6637048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6637169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6637450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6637587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6637762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6638063Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6638351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6638647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6638812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6638976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6639138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6639208Z ok (2.173s)
2022-11-23T01:19:07.6639214Z 
2022-11-23T01:19:07.6639412Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6639485Z Ran 1 test in 2.173s
2022-11-23T01:19:07.6639490Z 
2022-11-23T01:19:07.6639536Z OK
2022-11-23T01:19:07.6639554Z 
2022-11-23T01:19:07.6639651Z Generating XML reports...
2022-11-23T01:19:07.6639996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010741.xml
2022-11-23T01:19:07.6640225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6640502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6640626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6640906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6641067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6641073Z 
2022-11-23T01:19:07.6641144Z Running tests...
2022-11-23T01:19:07.6641326Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6641565Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3592
2022-11-23T01:19:07.6641720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3593
2022-11-23T01:19:07.6641872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3594
2022-11-23T01:19:07.6642143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6642265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6642544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6642680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6642856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6643116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6643237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6643515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6643651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6643828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6644098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6644222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6644501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6644623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6644799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6645097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6645393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6645685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6645852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6646013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6646175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6646375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6646534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6646708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6647003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6647295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6647623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6647868Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6648112Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6648358Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6648603Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6649014Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6649019Z 
2022-11-23T01:19:07.6649260Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6649515Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6649907Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6650153Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6650390Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6650629Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6650876Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6651281Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6651289Z 
2022-11-23T01:19:07.6651540Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6651789Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6652027Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6652263Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6652501Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6652739Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6652984Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6653223Z STAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6653477Z STAGE:2022-11-23 01:07:47 3594:3594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6653926Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:47 3593:3593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6653933Z 
2022-11-23T01:19:07.6654183Z STAGE:2022-11-23 01:07:47 3592:3592 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6654237Z ok (2.275s)
2022-11-23T01:19:07.6654255Z 
2022-11-23T01:19:07.6654440Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6654514Z Ran 1 test in 2.275s
2022-11-23T01:19:07.6654519Z 
2022-11-23T01:19:07.6654578Z OK
2022-11-23T01:19:07.6654584Z 
2022-11-23T01:19:07.6654667Z Generating XML reports...
2022-11-23T01:19:07.6655135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010745.xml
2022-11-23T01:19:07.6655426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6655699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6655826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6656092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6656229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6656234Z 
2022-11-23T01:19:07.6656303Z Running tests...
2022-11-23T01:19:07.6656500Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6656737Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3657
2022-11-23T01:19:07.6656896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3658
2022-11-23T01:19:07.6657049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3659
2022-11-23T01:19:07.6657324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6657448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6657714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6657847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6658025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6658301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6658425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6658699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6658832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6659010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6659267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6659388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6659668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6659800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6659978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6660280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6660577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6660906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6661072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6661224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6661387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6661561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6661765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6661936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6662232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6662531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6662824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6663069Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6663301Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6663541Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6663788Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6664030Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6664284Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6664526Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6664778Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6665031Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6665266Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6665648Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6665674Z 
2022-11-23T01:19:07.6665902Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6666312Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6666318Z 
2022-11-23T01:19:07.6666570Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6666822Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6667073Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6667310Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6667550Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6667785Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6668059Z STAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6668284Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6668705Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:07:51 3658:3658 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6668711Z 
2022-11-23T01:19:07.6668963Z STAGE:2022-11-23 01:07:51 3657:3657 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6669212Z STAGE:2022-11-23 01:07:51 3659:3659 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6669305Z ok (2.373s)
2022-11-23T01:19:07.6669310Z 
2022-11-23T01:19:07.6669506Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6669582Z Ran 1 test in 2.374s
2022-11-23T01:19:07.6669588Z 
2022-11-23T01:19:07.6669647Z OK
2022-11-23T01:19:07.6669652Z 
2022-11-23T01:19:07.6669737Z Generating XML reports...
2022-11-23T01:19:07.6670078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010749.xml
2022-11-23T01:19:07.6670297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6670571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6670696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6670978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6671117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6671122Z 
2022-11-23T01:19:07.6671194Z Running tests...
2022-11-23T01:19:07.6671390Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6671632Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3722
2022-11-23T01:19:07.6671773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3723
2022-11-23T01:19:07.6671926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3724
2022-11-23T01:19:07.6672200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6672323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6672608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6672747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6672923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6673197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6673321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6673587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6673721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6673897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6674197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6674469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6674592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6674937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6675071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6675245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6675532Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6675828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6676023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6676187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6676350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6676525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6676703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6676876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6677175Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6677458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6677753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6677996Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6678243Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6678483Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6678727Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6678971Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6679209Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6679463Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6679707Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6679960Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6680200Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6680437Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6680674Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6680916Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6681154Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6681395Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6681634Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6681885Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6682166Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6682405Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6682799Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6682805Z 
2022-11-23T01:19:07.6683046Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6683286Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6683729Z STAGE:2022-11-23 01:07:55 3723:3723 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6683736Z 
2022-11-23T01:19:07.6683988Z STAGE:2022-11-23 01:07:55 3722:3722 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6684240Z STAGE:2022-11-23 01:07:55 3724:3724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6684306Z ok (2.375s)
2022-11-23T01:19:07.6684311Z 
2022-11-23T01:19:07.6684493Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6684568Z Ran 1 test in 2.375s
2022-11-23T01:19:07.6684574Z 
2022-11-23T01:19:07.6684631Z OK
2022-11-23T01:19:07.6684636Z 
2022-11-23T01:19:07.6684719Z Generating XML reports...
2022-11-23T01:19:07.6685062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010753.xml
2022-11-23T01:19:07.6685296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6685567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6685693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6685961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6686095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6686100Z 
2022-11-23T01:19:07.6686171Z Running tests...
2022-11-23T01:19:07.6686366Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6686603Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3787
2022-11-23T01:19:07.6686759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3788
2022-11-23T01:19:07.6686910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3789
2022-11-23T01:19:07.6687184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6687307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6687574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6687708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6687884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6688154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6688279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6688556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6688690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6688893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6689192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6689448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6689569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6689945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6690113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6690291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6690591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6690889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6691053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6691216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6691361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6691535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6691711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6691883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6692181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6692478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6692769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6693020Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6693263Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6693488Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6693735Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6693975Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6694214Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6694466Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6694715Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6695089Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6695335Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6695938Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6695953Z 
2022-11-23T01:19:07.6696195Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6696503Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6696762Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6697005Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6697254Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6697502Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6697944Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6697949Z 
2022-11-23T01:19:07.6698189Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6698431Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6698675Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6698902Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6699155Z STAGE:2022-11-23 01:07:59 3787:3787 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6699404Z STAGE:2022-11-23 01:07:59 3788:3788 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6699648Z STAGE:2022-11-23 01:07:59 3789:3789 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6699711Z ok (2.174s)
2022-11-23T01:19:07.6699717Z 
2022-11-23T01:19:07.6699915Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6699992Z Ran 1 test in 2.174s
2022-11-23T01:19:07.6699997Z 
2022-11-23T01:19:07.6700058Z OK
2022-11-23T01:19:07.6700065Z 
2022-11-23T01:19:07.6700136Z Generating XML reports...
2022-11-23T01:19:07.6700477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010758.xml
2022-11-23T01:19:07.6700710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6700986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6701109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6701395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6701531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6701536Z 
2022-11-23T01:19:07.6701608Z Running tests...
2022-11-23T01:19:07.6701803Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6702025Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3852
2022-11-23T01:19:07.6702181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3853
2022-11-23T01:19:07.6702334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3854
2022-11-23T01:19:07.6702606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6702730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6703011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6703146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6703322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6703623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6703733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6704016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6704152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6704330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6704635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6704756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6705040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6705176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6705338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6705638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6705936Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6706229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6706400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6706564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6706739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6706912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6707072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6707231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6707526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6707819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6708113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6708356Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6708605Z STAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6709013Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6709019Z 
2022-11-23T01:19:07.6709275Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6709525Z STAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6709765Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6709990Z STAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6710235Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6710503Z STAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6710759Z STAGE:2022-11-23 01:08:03 3853:3853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6711011Z STAGE:2022-11-23 01:08:03 3854:3854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6711078Z ok (2.282s)
2022-11-23T01:19:07.6711083Z 
2022-11-23T01:19:07.6711281Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6711355Z Ran 1 test in 2.283s
2022-11-23T01:19:07.6711361Z 
2022-11-23T01:19:07.6711453Z OK
2022-11-23T01:19:07.6711458Z 
2022-11-23T01:19:07.6711529Z Generating XML reports...
2022-11-23T01:19:07.6711872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010801.xml
2022-11-23T01:19:07.6712110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6712387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6712512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6712793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6712927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6712933Z 
2022-11-23T01:19:07.6713004Z Running tests...
2022-11-23T01:19:07.6713187Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6713423Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3913
2022-11-23T01:19:07.6713578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3914
2022-11-23T01:19:07.6713729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3915
2022-11-23T01:19:07.6714007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6714130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6714411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6714545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6714719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6714981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6715103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6715383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6715519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6715697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6715965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6716086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6716368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6716490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6716669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6716964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6717291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6717590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6717750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6717915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6718077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6718281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6718440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6718611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6718908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6719202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6719447Z STAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6719736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6719978Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6720227Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6720467Z STAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6720723Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6720965Z STAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6721202Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6721438Z STAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6721843Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6721852Z 
2022-11-23T01:19:07.6722105Z STAGE:2022-11-23 01:08:07 3915:3915 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6722357Z STAGE:2022-11-23 01:08:07 3914:3914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6722424Z ok (2.273s)
2022-11-23T01:19:07.6722429Z 
2022-11-23T01:19:07.6722627Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6722702Z Ran 1 test in 2.273s
2022-11-23T01:19:07.6722709Z 
2022-11-23T01:19:07.6722756Z OK
2022-11-23T01:19:07.6722761Z 
2022-11-23T01:19:07.6722844Z Generating XML reports...
2022-11-23T01:19:07.6723187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010805.xml
2022-11-23T01:19:07.6723417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6723696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6723818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6724101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6724265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6724271Z 
2022-11-23T01:19:07.6724344Z Running tests...
2022-11-23T01:19:07.6724526Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6724767Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3974
2022-11-23T01:19:07.6724922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3975
2022-11-23T01:19:07.6725074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3976
2022-11-23T01:19:07.6725375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6725497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6725775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6725910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6726073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6726345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6726469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6726748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6726883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6727060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6727333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6727458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6727735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6727856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6728033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6728329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6728627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6728928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6729092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6729259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6729420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6729593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6729838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6730013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6730315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6730610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6730856Z STAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6731135Z STAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6731383Z STAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6731623Z STAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6732050Z STAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6732086Z 
2022-11-23T01:19:07.6732479Z STAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6732485Z 
2022-11-23T01:19:07.6732715Z STAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6732960Z STAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6733212Z STAGE:2022-11-23 01:08:11 3976:3976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6733460Z STAGE:2022-11-23 01:08:11 3975:3975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6733760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6733828Z ok (2.173s)
2022-11-23T01:19:07.6733836Z 
2022-11-23T01:19:07.6734032Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6734106Z Ran 1 test in 2.174s
2022-11-23T01:19:07.6734111Z 
2022-11-23T01:19:07.6734171Z OK
2022-11-23T01:19:07.6734176Z 
2022-11-23T01:19:07.6734247Z Generating XML reports...
2022-11-23T01:19:07.6734593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010809.xml
2022-11-23T01:19:07.6734825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6735233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6735359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6735644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6735782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6735790Z 
2022-11-23T01:19:07.6735861Z Running tests...
2022-11-23T01:19:07.6736056Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6736277Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4035
2022-11-23T01:19:07.6736434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4036
2022-11-23T01:19:07.6736587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4037
2022-11-23T01:19:07.6736860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6736985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6737263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6737401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6737578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6737838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6738012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6738292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6738427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6738603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6738901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6739173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6739337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6739616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6739738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6739912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6740209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6740505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6740670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6740837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6741009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6741182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6741346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6741501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6741799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6742094Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6742388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6742637Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6742879Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6743124Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6743365Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6743620Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6743863Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6744103Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6744342Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6744587Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6744828Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6745112Z STAGE:2022-11-23 01:08:15 4036:4036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6745367Z STAGE:2022-11-23 01:08:15 4037:4037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6745434Z ok (2.375s)
2022-11-23T01:19:07.6745439Z 
2022-11-23T01:19:07.6745633Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6745695Z Ran 1 test in 2.375s
2022-11-23T01:19:07.6745700Z 
2022-11-23T01:19:07.6745760Z OK
2022-11-23T01:19:07.6745765Z 
2022-11-23T01:19:07.6745848Z Generating XML reports...
2022-11-23T01:19:07.6746191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010813.xml
2022-11-23T01:19:07.6746463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6746739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6746866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6747147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6747269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6747288Z 
2022-11-23T01:19:07.6747347Z Running tests...
2022-11-23T01:19:07.6747543Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6747771Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4096
2022-11-23T01:19:07.6747928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4097
2022-11-23T01:19:07.6748080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4098
2022-11-23T01:19:07.6748351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6748475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6748757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6748879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6749055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6749325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6749449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6749733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6749868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6750046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6750317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6750425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6750704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6750840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6751014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6751313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6751611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6751931Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6752099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6752256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6752403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6752649Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6752891Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6753163Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6753406Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6753646Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6753882Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6754136Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6754389Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6754627Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6754864Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6755108Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6755344Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6755911Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6755917Z 
2022-11-23T01:19:07.6755921Z 
2022-11-23T01:19:07.6756175Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6756597Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6756604Z 
2022-11-23T01:19:07.6756841Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6757081Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6757318Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6757727Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6757971Z STAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6757976Z 
2022-11-23T01:19:07.6758215Z STAGE:2022-11-23 01:08:19 4097:4097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6758641Z STAGE:2022-11-23 01:08:19 4096:4096 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:19 4098:4098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6758646Z 
2022-11-23T01:19:07.6758713Z ok (2.175s)
2022-11-23T01:19:07.6758718Z 
2022-11-23T01:19:07.6758914Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6759018Z Ran 1 test in 2.175s
2022-11-23T01:19:07.6759023Z 
2022-11-23T01:19:07.6759083Z OK
2022-11-23T01:19:07.6759088Z 
2022-11-23T01:19:07.6759171Z Generating XML reports...
2022-11-23T01:19:07.6759513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010817.xml
2022-11-23T01:19:07.6759747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6760011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6760169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6760449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6760585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6760591Z 
2022-11-23T01:19:07.6760664Z Running tests...
2022-11-23T01:19:07.6760857Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6761083Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4152
2022-11-23T01:19:07.6761238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4153
2022-11-23T01:19:07.6761389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4154
2022-11-23T01:19:07.6761651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6761777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6762059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6762196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6762374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6762671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6762943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6763063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6763342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6763465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6763640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6763910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6764033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6764350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6764483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6764658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6764958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6765241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6765405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6765649Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6765838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6765995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6766236Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6766477Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6766723Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6766991Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6767222Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6767473Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6767729Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6767978Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6768218Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6768457Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6768691Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6768935Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6769342Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6769348Z 
2022-11-23T01:19:07.6769602Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6769938Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6770196Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6770436Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6770675Z STAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6770915Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6771158Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6771565Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6771571Z 
2022-11-23T01:19:07.6771826Z STAGE:2022-11-23 01:08:23 4153:4153 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6772249Z STAGE:2022-11-23 01:08:23 4154:4154 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:23 4152:4152 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6772254Z 
2022-11-23T01:19:07.6772321Z ok (2.272s)
2022-11-23T01:19:07.6772326Z 
2022-11-23T01:19:07.6772510Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6772586Z Ran 1 test in 2.272s
2022-11-23T01:19:07.6772591Z 
2022-11-23T01:19:07.6772651Z OK
2022-11-23T01:19:07.6772657Z 
2022-11-23T01:19:07.6772741Z Generating XML reports...
2022-11-23T01:19:07.6773119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010821.xml
2022-11-23T01:19:07.6773349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6773625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6773750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6774033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6774156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6774188Z 
2022-11-23T01:19:07.6774261Z Running tests...
2022-11-23T01:19:07.6774460Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6774692Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4208
2022-11-23T01:19:07.6774946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4209
2022-11-23T01:19:07.6775142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4210
2022-11-23T01:19:07.6775421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6775546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6775813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6775949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6776129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6776402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6776527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6776812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6776946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6777124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6777397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6777507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6777788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6777921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6778097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6778400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6778695Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6778989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6779158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6779324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6779475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6779570Z skip: CUDA is not available. (2.074s)
2022-11-23T01:19:07.6779576Z 
2022-11-23T01:19:07.6779772Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6779899Z Ran 1 test in 2.074s
2022-11-23T01:19:07.6779905Z 
2022-11-23T01:19:07.6779978Z OK (skipped=1)
2022-11-23T01:19:07.6779983Z 
2022-11-23T01:19:07.6780068Z Generating XML reports...
2022-11-23T01:19:07.6780410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010825.xml
2022-11-23T01:19:07.6780643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6780905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6781096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6781378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6781511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6781516Z 
2022-11-23T01:19:07.6781589Z Running tests...
2022-11-23T01:19:07.6781786Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6782030Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4261
2022-11-23T01:19:07.6782190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4262
2022-11-23T01:19:07.6782344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4263
2022-11-23T01:19:07.6782604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6782730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6783011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6783146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6783326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6783596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6783718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6783999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6784133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6784297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6784568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6784689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6784969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6785101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6785275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6785570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6785867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6786160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6786316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6786479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6786672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6786768Z skip: CUDA is not available. (2.174s)
2022-11-23T01:19:07.6786773Z 
2022-11-23T01:19:07.6786966Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6787040Z Ran 1 test in 2.174s
2022-11-23T01:19:07.6787045Z 
2022-11-23T01:19:07.6787115Z OK (skipped=1)
2022-11-23T01:19:07.6787119Z 
2022-11-23T01:19:07.6787202Z Generating XML reports...
2022-11-23T01:19:07.6787529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010829.xml
2022-11-23T01:19:07.6787790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6788063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6788187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6788470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6788605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6788610Z 
2022-11-23T01:19:07.6788681Z Running tests...
2022-11-23T01:19:07.6788873Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6789105Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4314
2022-11-23T01:19:07.6789250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4315
2022-11-23T01:19:07.6789402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4316
2022-11-23T01:19:07.6789676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6789798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6790080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6790212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6790387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6790658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6790767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6791048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6791180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6791356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6791656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6791925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6792047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6792325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6792458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6792624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6792921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6793248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6793413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6793574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6793732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6793980Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6794224Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6794508Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6794738Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6795153Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6795159Z 
2022-11-23T01:19:07.6795414Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6795835Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6795841Z 
2022-11-23T01:19:07.6796079Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6796319Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6796558Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6796801Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6797056Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6797459Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6797465Z 
2022-11-23T01:19:07.6797888Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6797895Z 
2022-11-23T01:19:07.6798133Z STAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6798356Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6798594Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6798837Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6799238Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6799244Z 
2022-11-23T01:19:07.6799496Z STAGE:2022-11-23 01:08:35 4315:4315 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6799916Z STAGE:2022-11-23 01:08:35 4316:4316 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:35 4314:4314 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6799924Z 
2022-11-23T01:19:07.6799990Z ok (2.177s)
2022-11-23T01:19:07.6799995Z 
2022-11-23T01:19:07.6800191Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6800302Z Ran 1 test in 2.177s
2022-11-23T01:19:07.6800308Z 
2022-11-23T01:19:07.6800368Z OK
2022-11-23T01:19:07.6800372Z 
2022-11-23T01:19:07.6800443Z Generating XML reports...
2022-11-23T01:19:07.6800785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010833.xml
2022-11-23T01:19:07.6801017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6801292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6801443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6801724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6801852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6801857Z 
2022-11-23T01:19:07.6801934Z Running tests...
2022-11-23T01:19:07.6802133Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6802368Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4370
2022-11-23T01:19:07.6802521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4371
2022-11-23T01:19:07.6802673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4372
2022-11-23T01:19:07.6802945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6803071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6803340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6803474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6803653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6803922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6804042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6804321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6804455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6804629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6804889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6805009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6805287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6805419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6805594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6805893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6806191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6806484Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6806648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6806795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6806990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6807087Z skip: CUDA is not available. (2.172s)
2022-11-23T01:19:07.6807092Z 
2022-11-23T01:19:07.6807285Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6807359Z Ran 1 test in 2.172s
2022-11-23T01:19:07.6807365Z 
2022-11-23T01:19:07.6807434Z OK (skipped=1)
2022-11-23T01:19:07.6807439Z 
2022-11-23T01:19:07.6807521Z Generating XML reports...
2022-11-23T01:19:07.6807861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010837.xml
2022-11-23T01:19:07.6808121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6808380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6808502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6808786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6808921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6808927Z 
2022-11-23T01:19:07.6808998Z Running tests...
2022-11-23T01:19:07.6809191Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6809418Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4423
2022-11-23T01:19:07.6809569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4424
2022-11-23T01:19:07.6809859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4425
2022-11-23T01:19:07.6810135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6810257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6810541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6810675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6810851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6811119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6811239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6811520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6811641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6811816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6812087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6812209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6812485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6812615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6812787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6813084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6813381Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6813697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6813861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6814022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6814269Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6814512Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6814673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6815036Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6815280Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6815532Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6815763Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6815999Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6816248Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6816501Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6816739Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6817133Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6817139Z 
2022-11-23T01:19:07.6817543Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6817550Z 
2022-11-23T01:19:07.6817788Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6818039Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6818288Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6818527Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6818764Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6819002Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6819241Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6819645Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6819652Z 
2022-11-23T01:19:07.6819895Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6820145Z STAGE:2022-11-23 01:08:43 4424:4424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6820389Z STAGE:2022-11-23 01:08:43 4425:4425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6820640Z STAGE:2022-11-23 01:08:43 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6820694Z ok (2.272s)
2022-11-23T01:19:07.6820711Z 
2022-11-23T01:19:07.6820892Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6820969Z Ran 1 test in 2.272s
2022-11-23T01:19:07.6821019Z 
2022-11-23T01:19:07.6821081Z OK
2022-11-23T01:19:07.6821086Z 
2022-11-23T01:19:07.6821176Z Generating XML reports...
2022-11-23T01:19:07.6821516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010841.xml
2022-11-23T01:19:07.6821750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6822026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6822150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6822463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6822598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6822603Z 
2022-11-23T01:19:07.6822673Z Running tests...
2022-11-23T01:19:07.6822871Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6823105Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4479
2022-11-23T01:19:07.6823258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4480
2022-11-23T01:19:07.6823409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4481
2022-11-23T01:19:07.6823683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6823809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6824075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6824206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6824385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6824658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6824779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6825060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6825194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6825368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6825628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6825749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6826026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6826159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6826335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6826637Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6826932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6827226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6827391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6827624Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6827813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6827970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6828213Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6828452Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6828698Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6828937Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6829207Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6829459Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6829701Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6829953Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6830189Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6830425Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6830662Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6830903Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6831143Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6831379Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6831631Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6831871Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6832123Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6832360Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6832599Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6832836Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6833238Z STAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6833244Z 
2022-11-23T01:19:07.6833486Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6833907Z STAGE:2022-11-23 01:08:47 4481:4481 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:47 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6833912Z 
2022-11-23T01:19:07.6834160Z STAGE:2022-11-23 01:08:47 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6834226Z ok (2.272s)
2022-11-23T01:19:07.6834231Z 
2022-11-23T01:19:07.6834414Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6834489Z Ran 1 test in 2.272s
2022-11-23T01:19:07.6834494Z 
2022-11-23T01:19:07.6834553Z OK
2022-11-23T01:19:07.6834558Z 
2022-11-23T01:19:07.6834641Z Generating XML reports...
2022-11-23T01:19:07.6834982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010845.xml
2022-11-23T01:19:07.6835241Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6835516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6835638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6835905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6836038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6836069Z 
2022-11-23T01:19:07.6836141Z Running tests...
2022-11-23T01:19:07.6836335Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6836572Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4535
2022-11-23T01:19:07.6836727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4536
2022-11-23T01:19:07.6836880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4537
2022-11-23T01:19:07.6837154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6837274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6837542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6837673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6837849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6838120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6838240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6838523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6838655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6838829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6839098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6839209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6839488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6839622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6839798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6840101Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6840398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6840692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6840857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6841099Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6841248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6841411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6841651Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6841921Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6842167Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6842574Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6842580Z 
2022-11-23T01:19:07.6842831Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6843299Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6843305Z 
2022-11-23T01:19:07.6843542Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6843782Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6844004Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6844245Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6844482Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6844732Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6844985Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6845224Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6845473Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6845708Z STAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6845943Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6846167Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.6846569Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6846577Z 
2022-11-23T01:19:07.6846817Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.6847239Z STAGE:2022-11-23 01:08:51 4537:4537 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:08:51 4535:4535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6847248Z 
2022-11-23T01:19:07.6847499Z STAGE:2022-11-23 01:08:51 4536:4536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.6847566Z ok (2.275s)
2022-11-23T01:19:07.6847571Z 
2022-11-23T01:19:07.6847768Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6847842Z Ran 1 test in 2.276s
2022-11-23T01:19:07.6847848Z 
2022-11-23T01:19:07.6847903Z OK
2022-11-23T01:19:07.6847909Z 
2022-11-23T01:19:07.6847991Z Generating XML reports...
2022-11-23T01:19:07.6848320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010849.xml
2022-11-23T01:19:07.6848552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6848824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6848945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6849253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6849388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6849394Z 
2022-11-23T01:19:07.6849464Z Running tests...
2022-11-23T01:19:07.6849657Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6849955Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4591
2022-11-23T01:19:07.6850134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4592
2022-11-23T01:19:07.6850286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4593
2022-11-23T01:19:07.6850562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6850686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6850964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6851094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6851271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6851545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6851654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6851936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6852068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6852243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6852516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6852637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6852916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6853046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6853220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6853511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6853806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6854105Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6854268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6854435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6854596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6854689Z skip: CUDA is not available. (2.175s)
2022-11-23T01:19:07.6854695Z 
2022-11-23T01:19:07.6854999Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6855076Z Ran 1 test in 2.176s
2022-11-23T01:19:07.6855082Z 
2022-11-23T01:19:07.6855140Z OK (skipped=1)
2022-11-23T01:19:07.6855145Z 
2022-11-23T01:19:07.6855227Z Generating XML reports...
2022-11-23T01:19:07.6855568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010853.xml
2022-11-23T01:19:07.6855845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6856119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6856241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6856519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6856652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6856657Z 
2022-11-23T01:19:07.6856752Z Running tests...
2022-11-23T01:19:07.6856946Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6857183Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4644
2022-11-23T01:19:07.6857337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4645
2022-11-23T01:19:07.6857491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4646
2022-11-23T01:19:07.6857763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6857883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6858160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6858294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6858460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6858730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6858850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6859129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6859263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6859439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6859710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6859830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6860091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6860228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6860404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6860701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6860999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6861291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6861453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6861615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6861780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6861862Z skip: CUDA is not available. (2.174s)
2022-11-23T01:19:07.6861878Z 
2022-11-23T01:19:07.6862061Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6862133Z Ran 1 test in 2.174s
2022-11-23T01:19:07.6862138Z 
2022-11-23T01:19:07.6862238Z OK (skipped=1)
2022-11-23T01:19:07.6862244Z 
2022-11-23T01:19:07.6862328Z Generating XML reports...
2022-11-23T01:19:07.6862667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010857.xml
2022-11-23T01:19:07.6862899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6863170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6863290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6863587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6863720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6863725Z 
2022-11-23T01:19:07.6863795Z Running tests...
2022-11-23T01:19:07.6863992Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6864234Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4697
2022-11-23T01:19:07.6864388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4698
2022-11-23T01:19:07.6864541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4699
2022-11-23T01:19:07.6864812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6864935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6865202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6865336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6865511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6865782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6865901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6866180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6866313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6866488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6866745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6866863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6867138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6867273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6867447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6867745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6868043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6868331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6868497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6868643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6868837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6868933Z skip: CUDA is not available. (2.273s)
2022-11-23T01:19:07.6868940Z 
2022-11-23T01:19:07.6869136Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6869210Z Ran 1 test in 2.274s
2022-11-23T01:19:07.6869216Z 
2022-11-23T01:19:07.6869285Z OK (skipped=1)
2022-11-23T01:19:07.6869291Z 
2022-11-23T01:19:07.6869374Z Generating XML reports...
2022-11-23T01:19:07.6869717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010901.xml
2022-11-23T01:19:07.6869976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6870235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6870356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6870636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6870768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6870773Z 
2022-11-23T01:19:07.6870843Z Running tests...
2022-11-23T01:19:07.6871036Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6871210Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:19:07.6871215Z 
2022-11-23T01:19:07.6871406Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6871469Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6871486Z 
2022-11-23T01:19:07.6871543Z OK (skipped=1)
2022-11-23T01:19:07.6871548Z 
2022-11-23T01:19:07.6871628Z Generating XML reports...
2022-11-23T01:19:07.6871969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010905.xml
2022-11-23T01:19:07.6872203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6872474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6872597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6872880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6873013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6873019Z 
2022-11-23T01:19:07.6873079Z Running tests...
2022-11-23T01:19:07.6873273Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6873454Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:19:07.6873459Z 
2022-11-23T01:19:07.6873649Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6873726Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6873731Z 
2022-11-23T01:19:07.6873800Z OK (skipped=1)
2022-11-23T01:19:07.6873805Z 
2022-11-23T01:19:07.6873886Z Generating XML reports...
2022-11-23T01:19:07.6874222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010906.xml
2022-11-23T01:19:07.6874453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6874711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6874833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6875112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6875242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6875248Z 
2022-11-23T01:19:07.6875320Z Running tests...
2022-11-23T01:19:07.6875543Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6875729Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-11-23T01:19:07.6875734Z 
2022-11-23T01:19:07.6875922Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6875996Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6876002Z 
2022-11-23T01:19:07.6876058Z OK (skipped=1)
2022-11-23T01:19:07.6876063Z 
2022-11-23T01:19:07.6876142Z Generating XML reports...
2022-11-23T01:19:07.6876507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010908.xml
2022-11-23T01:19:07.6876735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6877007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6877129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6877408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6877542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6877547Z 
2022-11-23T01:19:07.6877606Z Running tests...
2022-11-23T01:19:07.6877798Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6877992Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:19:07.6877999Z 
2022-11-23T01:19:07.6878188Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6878260Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6878265Z 
2022-11-23T01:19:07.6878334Z OK (skipped=1)
2022-11-23T01:19:07.6878339Z 
2022-11-23T01:19:07.6878420Z Generating XML reports...
2022-11-23T01:19:07.6878758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010910.xml
2022-11-23T01:19:07.6878987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6879246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6879366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6879642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6879776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6879781Z 
2022-11-23T01:19:07.6879851Z Running tests...
2022-11-23T01:19:07.6880038Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6880225Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:19:07.6880230Z 
2022-11-23T01:19:07.6880417Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6880488Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6880493Z 
2022-11-23T01:19:07.6880550Z OK (skipped=1)
2022-11-23T01:19:07.6880555Z 
2022-11-23T01:19:07.6880635Z Generating XML reports...
2022-11-23T01:19:07.6880975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010911.xml
2022-11-23T01:19:07.6881205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6881475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6881593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6881874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6882034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6882040Z 
2022-11-23T01:19:07.6882110Z Running tests...
2022-11-23T01:19:07.6882287Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6882484Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-11-23T01:19:07.6882489Z 
2022-11-23T01:19:07.6882680Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6882752Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6882782Z 
2022-11-23T01:19:07.6882850Z OK (skipped=1)
2022-11-23T01:19:07.6882854Z 
2022-11-23T01:19:07.6882933Z Generating XML reports...
2022-11-23T01:19:07.6883270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010913.xml
2022-11-23T01:19:07.6883503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6883765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6883886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6884167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6884299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6884304Z 
2022-11-23T01:19:07.6884374Z Running tests...
2022-11-23T01:19:07.6884568Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6884749Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:19:07.6884755Z 
2022-11-23T01:19:07.6884942Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6885015Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6885019Z 
2022-11-23T01:19:07.6885079Z OK (skipped=1)
2022-11-23T01:19:07.6885084Z 
2022-11-23T01:19:07.6885166Z Generating XML reports...
2022-11-23T01:19:07.6885500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010914.xml
2022-11-23T01:19:07.6885732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6886004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6886123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6886403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6886535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6886540Z 
2022-11-23T01:19:07.6886610Z Running tests...
2022-11-23T01:19:07.6886791Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6886989Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:19:07.6886995Z 
2022-11-23T01:19:07.6887183Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6887254Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6887259Z 
2022-11-23T01:19:07.6887331Z OK (skipped=1)
2022-11-23T01:19:07.6887336Z 
2022-11-23T01:19:07.6887415Z Generating XML reports...
2022-11-23T01:19:07.6887752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010916.xml
2022-11-23T01:19:07.6887985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6888243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6888430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6888711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6888846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6888851Z 
2022-11-23T01:19:07.6888924Z Running tests...
2022-11-23T01:19:07.6889117Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6889323Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6889369Z 
2022-11-23T01:19:07.6889560Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6889631Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6889637Z 
2022-11-23T01:19:07.6889771Z OK (skipped=1)
2022-11-23T01:19:07.6889792Z 
2022-11-23T01:19:07.6889863Z Generating XML reports...
2022-11-23T01:19:07.6890204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010918.xml
2022-11-23T01:19:07.6890436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6890708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6890829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6891107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6891241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6891246Z 
2022-11-23T01:19:07.6891316Z Running tests...
2022-11-23T01:19:07.6891495Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6891714Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6891720Z 
2022-11-23T01:19:07.6891910Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6891982Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6891988Z 
2022-11-23T01:19:07.6892056Z OK (skipped=1)
2022-11-23T01:19:07.6892061Z 
2022-11-23T01:19:07.6892143Z Generating XML reports...
2022-11-23T01:19:07.6892483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010919.xml
2022-11-23T01:19:07.6892714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6892986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6893096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6893374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6893508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6893514Z 
2022-11-23T01:19:07.6893583Z Running tests...
2022-11-23T01:19:07.6893772Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6893982Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:19:07.6893988Z 
2022-11-23T01:19:07.6894177Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6894249Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6894256Z 
2022-11-23T01:19:07.6894322Z OK (skipped=1)
2022-11-23T01:19:07.6894327Z 
2022-11-23T01:19:07.6894396Z Generating XML reports...
2022-11-23T01:19:07.6894735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010921.xml
2022-11-23T01:19:07.6895127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6895407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6895526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6895803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6895936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6895941Z 
2022-11-23T01:19:07.6896010Z Running tests...
2022-11-23T01:19:07.6896225Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6896446Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:19:07.6896452Z 
2022-11-23T01:19:07.6896639Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6896714Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6896719Z 
2022-11-23T01:19:07.6896788Z OK (skipped=1)
2022-11-23T01:19:07.6896793Z 
2022-11-23T01:19:07.6896875Z Generating XML reports...
2022-11-23T01:19:07.6897210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010923.xml
2022-11-23T01:19:07.6897441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6897713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6897825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6898105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6898238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6898243Z 
2022-11-23T01:19:07.6898315Z Running tests...
2022-11-23T01:19:07.6898508Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6898723Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6898729Z 
2022-11-23T01:19:07.6898916Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6898988Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6898993Z 
2022-11-23T01:19:07.6899061Z OK (skipped=1)
2022-11-23T01:19:07.6899066Z 
2022-11-23T01:19:07.6899135Z Generating XML reports...
2022-11-23T01:19:07.6899474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010924.xml
2022-11-23T01:19:07.6899704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6899973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6900096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6900375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6900507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6900513Z 
2022-11-23T01:19:07.6900581Z Running tests...
2022-11-23T01:19:07.6900773Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6900984Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:19:07.6900992Z 
2022-11-23T01:19:07.6901184Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6901257Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6901262Z 
2022-11-23T01:19:07.6901334Z OK (skipped=1)
2022-11-23T01:19:07.6901339Z 
2022-11-23T01:19:07.6901447Z Generating XML reports...
2022-11-23T01:19:07.6901787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010926.xml
2022-11-23T01:19:07.6902020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6902291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6902412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6902679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6902839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6902844Z 
2022-11-23T01:19:07.6902915Z Running tests...
2022-11-23T01:19:07.6903109Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6903324Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6903330Z 
2022-11-23T01:19:07.6903520Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6903591Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6903596Z 
2022-11-23T01:19:07.6903663Z OK (skipped=1)
2022-11-23T01:19:07.6903669Z 
2022-11-23T01:19:07.6903738Z Generating XML reports...
2022-11-23T01:19:07.6904073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010927.xml
2022-11-23T01:19:07.6904303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6904574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6904696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6904975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6905106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6905112Z 
2022-11-23T01:19:07.6905183Z Running tests...
2022-11-23T01:19:07.6905374Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6905580Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:19:07.6905597Z 
2022-11-23T01:19:07.6905772Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6905847Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6905852Z 
2022-11-23T01:19:07.6905921Z OK (skipped=1)
2022-11-23T01:19:07.6905926Z 
2022-11-23T01:19:07.6906009Z Generating XML reports...
2022-11-23T01:19:07.6906345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010929.xml
2022-11-23T01:19:07.6906577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6906849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6906970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6907238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6907370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6907377Z 
2022-11-23T01:19:07.6907446Z Running tests...
2022-11-23T01:19:07.6907638Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6907848Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6907853Z 
2022-11-23T01:19:07.6908088Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6908163Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6908169Z 
2022-11-23T01:19:07.6908239Z OK (skipped=1)
2022-11-23T01:19:07.6908244Z 
2022-11-23T01:19:07.6908326Z Generating XML reports...
2022-11-23T01:19:07.6908652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010931.xml
2022-11-23T01:19:07.6908880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6909151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6909304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6909584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6909718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6909727Z 
2022-11-23T01:19:07.6909798Z Running tests...
2022-11-23T01:19:07.6909991Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6910197Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6910216Z 
2022-11-23T01:19:07.6910394Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6910466Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6910471Z 
2022-11-23T01:19:07.6910539Z OK (skipped=1)
2022-11-23T01:19:07.6910547Z 
2022-11-23T01:19:07.6910629Z Generating XML reports...
2022-11-23T01:19:07.6910964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010932.xml
2022-11-23T01:19:07.6911194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6911467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6911589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6911858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6911991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6911996Z 
2022-11-23T01:19:07.6912066Z Running tests...
2022-11-23T01:19:07.6912256Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6912475Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:19:07.6912481Z 
2022-11-23T01:19:07.6912672Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6912745Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6912750Z 
2022-11-23T01:19:07.6912818Z OK (skipped=1)
2022-11-23T01:19:07.6912825Z 
2022-11-23T01:19:07.6912907Z Generating XML reports...
2022-11-23T01:19:07.6913229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010934.xml
2022-11-23T01:19:07.6913458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6913731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6913852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6914138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6914272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6914277Z 
2022-11-23T01:19:07.6914347Z Running tests...
2022-11-23T01:19:07.6914539Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6914791Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:19:07.6914798Z 
2022-11-23T01:19:07.6914978Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6915051Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6915056Z 
2022-11-23T01:19:07.6915125Z OK (skipped=1)
2022-11-23T01:19:07.6915130Z 
2022-11-23T01:19:07.6915212Z Generating XML reports...
2022-11-23T01:19:07.6915549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010935.xml
2022-11-23T01:19:07.6915809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6916082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6916203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6916484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6916604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6916609Z 
2022-11-23T01:19:07.6916678Z Running tests...
2022-11-23T01:19:07.6916870Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6917085Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6917092Z 
2022-11-23T01:19:07.6917281Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6917353Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6917358Z 
2022-11-23T01:19:07.6917429Z OK (skipped=1)
2022-11-23T01:19:07.6917434Z 
2022-11-23T01:19:07.6917516Z Generating XML reports...
2022-11-23T01:19:07.6917853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010937.xml
2022-11-23T01:19:07.6918070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6918342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6918464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6918742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6918874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6918879Z 
2022-11-23T01:19:07.6918951Z Running tests...
2022-11-23T01:19:07.6919144Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6919373Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:19:07.6919379Z 
2022-11-23T01:19:07.6919569Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6919630Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6919635Z 
2022-11-23T01:19:07.6919704Z OK (skipped=1)
2022-11-23T01:19:07.6919709Z 
2022-11-23T01:19:07.6919791Z Generating XML reports...
2022-11-23T01:19:07.6920124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010939.xml
2022-11-23T01:19:07.6920355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6920627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6920749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6921028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6921174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6921192Z 
2022-11-23T01:19:07.6921252Z Running tests...
2022-11-23T01:19:07.6921444Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6921658Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:19:07.6921664Z 
2022-11-23T01:19:07.6921856Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6921930Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6921963Z 
2022-11-23T01:19:07.6922033Z OK (skipped=1)
2022-11-23T01:19:07.6922038Z 
2022-11-23T01:19:07.6922120Z Generating XML reports...
2022-11-23T01:19:07.6922458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010940.xml
2022-11-23T01:19:07.6922677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6922947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6923069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6923352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6923485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6923490Z 
2022-11-23T01:19:07.6923561Z Running tests...
2022-11-23T01:19:07.6923757Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6923977Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:19:07.6923983Z 
2022-11-23T01:19:07.6924172Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6924234Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6924240Z 
2022-11-23T01:19:07.6924309Z OK (skipped=1)
2022-11-23T01:19:07.6924313Z 
2022-11-23T01:19:07.6924395Z Generating XML reports...
2022-11-23T01:19:07.6924734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010942.xml
2022-11-23T01:19:07.6924963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6925234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6925358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6925636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6925769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6925775Z 
2022-11-23T01:19:07.6925833Z Running tests...
2022-11-23T01:19:07.6926027Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6926260Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4990
2022-11-23T01:19:07.6926417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4991
2022-11-23T01:19:07.6926569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4992
2022-11-23T01:19:07.6926841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6926964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6927244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6927364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6927569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6927843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6927965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6928245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6928377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6928551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6928878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6929146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6929256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6929531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6929664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6929920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6930221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6930519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6930684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6930849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6931007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6931097Z skip: Need at least 2 CUDA devices (2.081s)
2022-11-23T01:19:07.6931102Z 
2022-11-23T01:19:07.6931296Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6931369Z Ran 1 test in 2.081s
2022-11-23T01:19:07.6931374Z 
2022-11-23T01:19:07.6931444Z OK (skipped=1)
2022-11-23T01:19:07.6931449Z 
2022-11-23T01:19:07.6931532Z Generating XML reports...
2022-11-23T01:19:07.6931871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010943.xml
2022-11-23T01:19:07.6932105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6932377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6932488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6932767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6932899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6932904Z 
2022-11-23T01:19:07.6932975Z Running tests...
2022-11-23T01:19:07.6933167Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6933482Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2022-11-23T01:19:07.6933490Z 
2022-11-23T01:19:07.6933682Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6933757Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6933762Z 
2022-11-23T01:19:07.6933831Z OK (skipped=1)
2022-11-23T01:19:07.6933835Z 
2022-11-23T01:19:07.6933905Z Generating XML reports...
2022-11-23T01:19:07.6934280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010947.xml
2022-11-23T01:19:07.6934513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6934783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6935069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6935356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6935490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6935554Z 
2022-11-23T01:19:07.6935627Z Running tests...
2022-11-23T01:19:07.6935822Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6936120Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2022-11-23T01:19:07.6936126Z 
2022-11-23T01:19:07.6936321Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6936394Z Ran 1 test in 0.002s
2022-11-23T01:19:07.6936399Z 
2022-11-23T01:19:07.6936468Z OK (skipped=1)
2022-11-23T01:19:07.6936473Z 
2022-11-23T01:19:07.6936556Z Generating XML reports...
2022-11-23T01:19:07.6936896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010949.xml
2022-11-23T01:19:07.6937129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6937406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6937528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6937794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6937931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6937936Z 
2022-11-23T01:19:07.6938006Z Running tests...
2022-11-23T01:19:07.6938198Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6938418Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5063
2022-11-23T01:19:07.6938572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5064
2022-11-23T01:19:07.6938723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5065
2022-11-23T01:19:07.6938994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6939104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6939382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6939516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6939695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6939993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6940260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6940381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6940662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6940793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6940954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6941265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6941388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6941668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6941802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6941976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6942276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6942600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6942765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6942919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6943083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6943150Z ok (3.576s)
2022-11-23T01:19:07.6943156Z 
2022-11-23T01:19:07.6943353Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6943426Z Ran 1 test in 3.576s
2022-11-23T01:19:07.6943432Z 
2022-11-23T01:19:07.6943490Z OK
2022-11-23T01:19:07.6943495Z 
2022-11-23T01:19:07.6943577Z Generating XML reports...
2022-11-23T01:19:07.6943918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010950.xml
2022-11-23T01:19:07.6944140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6944411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6944535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6944816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6944948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6944953Z 
2022-11-23T01:19:07.6945024Z Running tests...
2022-11-23T01:19:07.6945218Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6945443Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5116
2022-11-23T01:19:07.6945599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5117
2022-11-23T01:19:07.6945739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5118
2022-11-23T01:19:07.6946014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6946138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6946416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6946549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6946726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6946996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6947118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6947381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6947514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6947717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6947991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6948111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6948388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6948521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6948697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6949023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6949308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6949600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6949765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6949928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6950089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6950183Z skip: CUDA is not available. (2.175s)
2022-11-23T01:19:07.6950188Z 
2022-11-23T01:19:07.6950382Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6950458Z Ran 1 test in 2.175s
2022-11-23T01:19:07.6950463Z 
2022-11-23T01:19:07.6950533Z OK (skipped=1)
2022-11-23T01:19:07.6950538Z 
2022-11-23T01:19:07.6950607Z Generating XML reports...
2022-11-23T01:19:07.6950947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010956.xml
2022-11-23T01:19:07.6951182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6951456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6951579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6951859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6951994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6952001Z 
2022-11-23T01:19:07.6952073Z Running tests...
2022-11-23T01:19:07.6952256Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6952485Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5169
2022-11-23T01:19:07.6952642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5170
2022-11-23T01:19:07.6980173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5171
2022-11-23T01:19:07.6980622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6980745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6981023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6981150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6981326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6981594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6981707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6982113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6982242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6982408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6982673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6982786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6983104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6983228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6983396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6983692Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6983985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6984270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6984422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6984576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6984730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6984894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.6985059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.6985223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.6985509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6985803Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6986093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.6986150Z ok (3.579s)
2022-11-23T01:19:07.6986157Z 
2022-11-23T01:19:07.6986356Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6986432Z Ran 1 test in 3.579s
2022-11-23T01:19:07.6986438Z 
2022-11-23T01:19:07.6986495Z OK
2022-11-23T01:19:07.6986500Z 
2022-11-23T01:19:07.6986583Z Generating XML reports...
2022-11-23T01:19:07.6986931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011000.xml
2022-11-23T01:19:07.6987167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6987442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6987553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6987833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6987967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6987973Z 
2022-11-23T01:19:07.6988044Z Running tests...
2022-11-23T01:19:07.6988238Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6988505Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5231
2022-11-23T01:19:07.6988661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5232
2022-11-23T01:19:07.6988811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5233
2022-11-23T01:19:07.6989085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6989196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6989475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6989641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6989816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6990085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6990213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6990495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6990632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6990812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6991099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6991371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6991492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6991769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6991904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6992080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6992377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6992672Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6992833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.6992986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.6993147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.6993241Z skip: CUDA is not available. (2.172s)
2022-11-23T01:19:07.6993247Z 
2022-11-23T01:19:07.6993442Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6993514Z Ran 1 test in 2.172s
2022-11-23T01:19:07.6993519Z 
2022-11-23T01:19:07.6993590Z OK (skipped=1)
2022-11-23T01:19:07.6993595Z 
2022-11-23T01:19:07.6993679Z Generating XML reports...
2022-11-23T01:19:07.6994020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011005.xml
2022-11-23T01:19:07.6994236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.6994512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6994637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6994915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6995048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6995083Z 
2022-11-23T01:19:07.6995155Z Running tests...
2022-11-23T01:19:07.6995351Z ----------------------------------------------------------------------
2022-11-23T01:19:07.6995579Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5284
2022-11-23T01:19:07.6995732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5285
2022-11-23T01:19:07.6995872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5286
2022-11-23T01:19:07.6996144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6996296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6996577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6996712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6996886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.6997152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6997275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6997538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6997673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6997850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.6998147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6998416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.6998536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.6998815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.6998948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.6999122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.6999407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6999702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.6999865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7000031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7000191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7000364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7000536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7000705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7000999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7001281Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7001571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7001637Z ok (3.075s)
2022-11-23T01:19:07.7001669Z 
2022-11-23T01:19:07.7001867Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7001941Z Ran 1 test in 3.076s
2022-11-23T01:19:07.7001947Z 
2022-11-23T01:19:07.7002002Z OK
2022-11-23T01:19:07.7002007Z 
2022-11-23T01:19:07.7002088Z Generating XML reports...
2022-11-23T01:19:07.7002432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011009.xml
2022-11-23T01:19:07.7002663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7002954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7003076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7003354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7003490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7003495Z 
2022-11-23T01:19:07.7003566Z Running tests...
2022-11-23T01:19:07.7003760Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7003992Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5343
2022-11-23T01:19:07.7004146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5344
2022-11-23T01:19:07.7004284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5345
2022-11-23T01:19:07.7004559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7004681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7004962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7005098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7005272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7005542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7005663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7005941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7006065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7006239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7006540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7006813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7006935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7007211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7007342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7007516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7007813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7008098Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7008265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7008458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7008623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7008718Z skip: CUDA is not available. (2.073s)
2022-11-23T01:19:07.7008724Z 
2022-11-23T01:19:07.7008917Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7008991Z Ran 1 test in 2.073s
2022-11-23T01:19:07.7008996Z 
2022-11-23T01:19:07.7009065Z OK (skipped=1)
2022-11-23T01:19:07.7009071Z 
2022-11-23T01:19:07.7009182Z Generating XML reports...
2022-11-23T01:19:07.7009521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011014.xml
2022-11-23T01:19:07.7009848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7010127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7010250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7010530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7010665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7010670Z 
2022-11-23T01:19:07.7010740Z Running tests...
2022-11-23T01:19:07.7010932Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7011160Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5396
2022-11-23T01:19:07.7011317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5397
2022-11-23T01:19:07.7011467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5398
2022-11-23T01:19:07.7011741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7011861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7012142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7012275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7012451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7012721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7012833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7013115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7013250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7013424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7013695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7013815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7014095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7014228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7014395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7014692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7015220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7015517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7015681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7015844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7016004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7016175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7016415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7016576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7016871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7017162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7017450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7017518Z ok (3.377s)
2022-11-23T01:19:07.7017524Z 
2022-11-23T01:19:07.7017718Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7017795Z Ran 1 test in 3.377s
2022-11-23T01:19:07.7017802Z 
2022-11-23T01:19:07.7017860Z OK
2022-11-23T01:19:07.7017865Z 
2022-11-23T01:19:07.7017947Z Generating XML reports...
2022-11-23T01:19:07.7018277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011017.xml
2022-11-23T01:19:07.7018510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7018785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7018907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7019186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7019320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7019326Z 
2022-11-23T01:19:07.7019393Z Running tests...
2022-11-23T01:19:07.7019588Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7019896Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-11-23T01:19:07.7019902Z 
2022-11-23T01:19:07.7020079Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7020155Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7020161Z 
2022-11-23T01:19:07.7020231Z OK (skipped=1)
2022-11-23T01:19:07.7020236Z 
2022-11-23T01:19:07.7020317Z Generating XML reports...
2022-11-23T01:19:07.7020657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011023.xml
2022-11-23T01:19:07.7020886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7021160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7021283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7021564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7021686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7021692Z 
2022-11-23T01:19:07.7021789Z Running tests...
2022-11-23T01:19:07.7021984Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7022219Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5468
2022-11-23T01:19:07.7022373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5469
2022-11-23T01:19:07.7022524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5470
2022-11-23T01:19:07.7022795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7022945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7023225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7023348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7023526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7023800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7023922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7024200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7024332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7024509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7024806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7025063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7025185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7025462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7025598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7025772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7026067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7026359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7026520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7026684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7026835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7027008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7027180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7027355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7027648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7027944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7028232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7028299Z ok (7.181s)
2022-11-23T01:19:07.7028304Z 
2022-11-23T01:19:07.7028527Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7028590Z Ran 1 test in 7.182s
2022-11-23T01:19:07.7028595Z 
2022-11-23T01:19:07.7028654Z OK
2022-11-23T01:19:07.7028659Z 
2022-11-23T01:19:07.7028743Z Generating XML reports...
2022-11-23T01:19:07.7029088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011024.xml
2022-11-23T01:19:07.7029321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7029625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7029745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7030024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7030158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7030164Z 
2022-11-23T01:19:07.7030222Z Running tests...
2022-11-23T01:19:07.7030415Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7030653Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5527
2022-11-23T01:19:07.7030807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5528
2022-11-23T01:19:07.7030959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5529
2022-11-23T01:19:07.7031234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7031354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7031632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7031757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7031933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7032203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7032324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7032605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7032740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7032915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7033211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7033482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7033593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7033870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7034002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7034174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7034471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7034770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7034933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7035123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7035286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7035340Z ok (2.274s)
2022-11-23T01:19:07.7035346Z 
2022-11-23T01:19:07.7035540Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7035614Z Ran 1 test in 2.275s
2022-11-23T01:19:07.7035619Z 
2022-11-23T01:19:07.7035678Z OK
2022-11-23T01:19:07.7035683Z 
2022-11-23T01:19:07.7035763Z Generating XML reports...
2022-11-23T01:19:07.7036129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011033.xml
2022-11-23T01:19:07.7036359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7036631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7036743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7037020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7037151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7037157Z 
2022-11-23T01:19:07.7037227Z Running tests...
2022-11-23T01:19:07.7037422Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7037661Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5580
2022-11-23T01:19:07.7037816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5581
2022-11-23T01:19:07.7037967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5582
2022-11-23T01:19:07.7038237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7038347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7038626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7038759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7038936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7039206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7039330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7039609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7039742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7039908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7040207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7040478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7040599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7040873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7041005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7041177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7041473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7041794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7041947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7042109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7042271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7042337Z ok (2.178s)
2022-11-23T01:19:07.7042342Z 
2022-11-23T01:19:07.7042535Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7042636Z Ran 1 test in 2.179s
2022-11-23T01:19:07.7042642Z 
2022-11-23T01:19:07.7042702Z OK
2022-11-23T01:19:07.7042707Z 
2022-11-23T01:19:07.7042788Z Generating XML reports...
2022-11-23T01:19:07.7043133Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011037.xml
2022-11-23T01:19:07.7043356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7043625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7043747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7044027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7044160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7044168Z 
2022-11-23T01:19:07.7044236Z Running tests...
2022-11-23T01:19:07.7044431Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7044627Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7044632Z 
2022-11-23T01:19:07.7044823Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7044884Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7044889Z 
2022-11-23T01:19:07.7044958Z OK (skipped=1)
2022-11-23T01:19:07.7044963Z 
2022-11-23T01:19:07.7045047Z Generating XML reports...
2022-11-23T01:19:07.7045385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011041.xml
2022-11-23T01:19:07.7045615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7045889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7046013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7046292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7046412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7046429Z 
2022-11-23T01:19:07.7046489Z Running tests...
2022-11-23T01:19:07.7046681Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7046869Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7046874Z 
2022-11-23T01:19:07.7047061Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7047136Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7047141Z 
2022-11-23T01:19:07.7047210Z OK (skipped=1)
2022-11-23T01:19:07.7047215Z 
2022-11-23T01:19:07.7047299Z Generating XML reports...
2022-11-23T01:19:07.7047637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011043.xml
2022-11-23T01:19:07.7047854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7048152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7048276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7048555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7048689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7048695Z 
2022-11-23T01:19:07.7048764Z Running tests...
2022-11-23T01:19:07.7048960Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7049156Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7049200Z 
2022-11-23T01:19:07.7049392Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7049452Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7049458Z 
2022-11-23T01:19:07.7049526Z OK (skipped=1)
2022-11-23T01:19:07.7049532Z 
2022-11-23T01:19:07.7049619Z Generating XML reports...
2022-11-23T01:19:07.7050045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011044.xml
2022-11-23T01:19:07.7050278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7050549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7050673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7050952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7051077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7051094Z 
2022-11-23T01:19:07.7051152Z Running tests...
2022-11-23T01:19:07.7051344Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7051531Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7051537Z 
2022-11-23T01:19:07.7051723Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7051795Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7051801Z 
2022-11-23T01:19:07.7051870Z OK (skipped=1)
2022-11-23T01:19:07.7051875Z 
2022-11-23T01:19:07.7051957Z Generating XML reports...
2022-11-23T01:19:07.7052293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011046.xml
2022-11-23T01:19:07.7052511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7052782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7052902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7053181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7053319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7053325Z 
2022-11-23T01:19:07.7053395Z Running tests...
2022-11-23T01:19:07.7053588Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7053777Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7053783Z 
2022-11-23T01:19:07.7053972Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7054034Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7054039Z 
2022-11-23T01:19:07.7054108Z OK (skipped=1)
2022-11-23T01:19:07.7054113Z 
2022-11-23T01:19:07.7054193Z Generating XML reports...
2022-11-23T01:19:07.7054533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011047.xml
2022-11-23T01:19:07.7054797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7055160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7055282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7055561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7055695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7055701Z 
2022-11-23T01:19:07.7055821Z Running tests...
2022-11-23T01:19:07.7056016Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7056216Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7056222Z 
2022-11-23T01:19:07.7056413Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7056491Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7056496Z 
2022-11-23T01:19:07.7056566Z OK (skipped=1)
2022-11-23T01:19:07.7056571Z 
2022-11-23T01:19:07.7056653Z Generating XML reports...
2022-11-23T01:19:07.7056992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011049.xml
2022-11-23T01:19:07.7057213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7057487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7057614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7057894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7058028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7058033Z 
2022-11-23T01:19:07.7058106Z Running tests...
2022-11-23T01:19:07.7058298Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7058488Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7058493Z 
2022-11-23T01:19:07.7058684Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7058745Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7058763Z 
2022-11-23T01:19:07.7058820Z OK (skipped=1)
2022-11-23T01:19:07.7058825Z 
2022-11-23T01:19:07.7058906Z Generating XML reports...
2022-11-23T01:19:07.7059244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011051.xml
2022-11-23T01:19:07.7059474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7059747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7059870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7060147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7060278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7060283Z 
2022-11-23T01:19:07.7060341Z Running tests...
2022-11-23T01:19:07.7060533Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7060721Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:19:07.7060729Z 
2022-11-23T01:19:07.7060920Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7060994Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7061000Z 
2022-11-23T01:19:07.7061070Z OK (skipped=1)
2022-11-23T01:19:07.7061075Z 
2022-11-23T01:19:07.7061159Z Generating XML reports...
2022-11-23T01:19:07.7061531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011052.xml
2022-11-23T01:19:07.7061763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7062025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7062148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7062427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7062590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7062595Z 
2022-11-23T01:19:07.7062666Z Running tests...
2022-11-23T01:19:07.7062862Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7063089Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5713
2022-11-23T01:19:07.7063243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5714
2022-11-23T01:19:07.7063383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5715
2022-11-23T01:19:07.7063655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7063777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7064055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7064188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7064363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7064634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7064756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7065033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7065156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7065331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7065603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7065727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7066005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7066137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7066314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7066613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7066909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7067190Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7067356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7067522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7067683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7067928Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7068194Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7068435Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7068678Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7068906Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7069327Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7069359Z 
2022-11-23T01:19:07.7069614Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7069868Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7070108Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7070344Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7070578Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7071145Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7071153Z 
2022-11-23T01:19:07.7071158Z 
2022-11-23T01:19:07.7071582Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7071837Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7071842Z 
2022-11-23T01:19:07.7072075Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7072309Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7072539Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7072789Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7073026Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7073265Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7073501Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7073752Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7074000Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7074235Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7074468Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7074691Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7074937Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7075349Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7075381Z 
2022-11-23T01:19:07.7075622Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7075871Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7076122Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7076358Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7076593Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7076855Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7077096Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7077326Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7077575Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7077815Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7078067Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7078316Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7078554Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7078789Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7079027Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7079267Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7079505Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7079743Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7079978Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7080401Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7080409Z 
2022-11-23T01:19:07.7080646Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7080882Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7081117Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7081356Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7081605Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7081836Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7082089Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7082330Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7082585Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7082819Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7083080Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7083318Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7083559Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7083967Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7084001Z 
2022-11-23T01:19:07.7084242Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7084479Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7084729Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7084967Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7085201Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7085436Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7085680Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7085917Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7086169Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7086408Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7086646Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7086898Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7087135Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7087368Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7087602Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7087842Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7088420Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7088426Z 
2022-11-23T01:19:07.7088431Z 
2022-11-23T01:19:07.7088682Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7088931Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7089166Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7089389Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7089624Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7089935Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7090547Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7090554Z 
2022-11-23T01:19:07.7090558Z 
2022-11-23T01:19:07.7090809Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7091061Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7091299Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7091535Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7091800Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7092043Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7092271Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7092521Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7092770Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7093013Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7093262Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7093497Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7093732Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7093969Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7094208Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7094445Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7094685Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7095030Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7095284Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7095533Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7095771Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7096167Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7096173Z 
2022-11-23T01:19:07.7096409Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7096647Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7096880Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7097118Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7097366Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7097620Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7097858Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7098145Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7098382Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7098948Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7098954Z 
2022-11-23T01:19:07.7098958Z 
2022-11-23T01:19:07.7099245Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7099668Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7099674Z 
2022-11-23T01:19:07.7099910Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7100144Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7100367Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7100610Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7100860Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7101102Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7101342Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7101592Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7101842Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7102078Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7102314Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7102537Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7102776Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7103015Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7103246Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7103494Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7103747Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7103996Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7104232Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7104464Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7104686Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7104927Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7105174Z STAGE:2022-11-23 01:10:56 5715:5715 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7105605Z STAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7105612Z 
2022-11-23T01:19:07.7106035Z STAGE:2022-11-23 01:10:56 5714:5714 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:10:56 5713:5713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7106041Z 
2022-11-23T01:19:07.7106107Z ok (2.273s)
2022-11-23T01:19:07.7106112Z 
2022-11-23T01:19:07.7106307Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7106381Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7106427Z 
2022-11-23T01:19:07.7106487Z OK
2022-11-23T01:19:07.7106492Z 
2022-11-23T01:19:07.7106575Z Generating XML reports...
2022-11-23T01:19:07.7106904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011054.xml
2022-11-23T01:19:07.7107137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7107413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7107582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7107862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7107995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7108000Z 
2022-11-23T01:19:07.7108070Z Running tests...
2022-11-23T01:19:07.7108267Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7108956Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2022-11-23T01:19:07.7108962Z 
2022-11-23T01:19:07.7109153Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7109214Z Ran 1 test in 0.258s
2022-11-23T01:19:07.7109220Z 
2022-11-23T01:19:07.7109289Z OK (skipped=1)
2022-11-23T01:19:07.7109293Z 
2022-11-23T01:19:07.7109376Z Generating XML reports...
2022-11-23T01:19:07.7109717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011058.xml
2022-11-23T01:19:07.7109947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7110221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7110343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7110625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7110747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7110763Z 
2022-11-23T01:19:07.7110820Z Running tests...
2022-11-23T01:19:07.7111012Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7111246Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5779
2022-11-23T01:19:07.7111396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5780
2022-11-23T01:19:07.7111551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5781
2022-11-23T01:19:07.7111821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7111945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7112250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7112374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7112553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7112828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7112949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7113230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7113390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7113567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7113871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7114142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7114253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7114533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7114666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7114841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7115142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7115438Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7115601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7115766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7115916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7116090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7116264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7116436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7116733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7117025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7117311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7117556Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7117799Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7118035Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7118287Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7118525Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7118761Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7119007Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7119272Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7119528Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7119778Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7120013Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7120235Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7120501Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7120738Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7120991Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7121233Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7121482Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7121736Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7121970Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7122205Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7122431Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7122668Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7122921Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7123160Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7123396Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7123647Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7123896Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7124129Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7124367Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7124590Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7124831Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7125083Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7125320Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7125555Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7125979Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7125987Z 
2022-11-23T01:19:07.7126226Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7126460Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7126756Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7126983Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7127235Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7127477Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7127727Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7127971Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7128252Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7128488Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7128724Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7128957Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7129184Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7129433Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7129674Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7129980Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7130237Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7130489Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7130884Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7130890Z 
2022-11-23T01:19:07.7131124Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7131363Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7131780Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7131788Z 
2022-11-23T01:19:07.7132202Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7132207Z 
2022-11-23T01:19:07.7132458Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7132838Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7132856Z 
2022-11-23T01:19:07.7133078Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7133320Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7133574Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7133814Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7134062Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7134333Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7134585Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7134822Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7135157Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7135391Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7135676Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7135912Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7136162Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7136405Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7136654Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7136904Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7137139Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7137362Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7137596Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7137834Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7138070Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7138321Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7138732Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7138737Z 
2022-11-23T01:19:07.7138986Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7139224Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7139461Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7139684Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7139926Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7140177Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7140416Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7140665Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7140905Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7141156Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7141394Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7141628Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7141884Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7142129Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7142364Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7142612Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7142852Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7143099Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7143378Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7143614Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7143849Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7144073Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7144312Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7144559Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7144800Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7145048Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7145290Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7145538Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7145774Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7146008Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7146231Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7146470Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7146719Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7146955Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7147202Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7147441Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7147691Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7147928Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7148161Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7148383Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7148790Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7148799Z 
2022-11-23T01:19:07.7149041Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7149489Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7149495Z 
2022-11-23T01:19:07.7149746Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7149983Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7150218Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7150453Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7150722Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7150974Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7151371Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7151390Z 
2022-11-23T01:19:07.7151628Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7151880Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7152116Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7152350Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7152587Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7152829Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7153067Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7153321Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7153598Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7153835Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7154082Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7154316Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7154555Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7154788Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7155028Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7155268Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7155518Z STAGE:2022-11-23 01:11:02 5781:5781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7155759Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7155997Z STAGE:2022-11-23 01:11:02 5779:5779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7156248Z STAGE:2022-11-23 01:11:02 5780:5780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7156316Z ok (2.272s)
2022-11-23T01:19:07.7156321Z 
2022-11-23T01:19:07.7156517Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7156591Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7156597Z 
2022-11-23T01:19:07.7156657Z OK
2022-11-23T01:19:07.7156662Z 
2022-11-23T01:19:07.7156744Z Generating XML reports...
2022-11-23T01:19:07.7157112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011100.xml
2022-11-23T01:19:07.7157333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7157610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7157734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7158014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7158188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7158193Z 
2022-11-23T01:19:07.7158264Z Running tests...
2022-11-23T01:19:07.7158461Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7158695Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5844
2022-11-23T01:19:07.7158849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5845
2022-11-23T01:19:07.7158990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5846
2022-11-23T01:19:07.7159261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7159383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7159665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7159800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7159972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7160246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7160367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7160635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7160770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7160949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7161221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7161345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7161623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7161756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7161930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7162229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7162515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7162811Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7162974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7163139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7163300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7163474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7163677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7163850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7164147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7164428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7164672Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7164997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7165236Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7165482Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7165723Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7165974Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7166226Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7166464Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7166691Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7166930Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7167168Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7167593Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7167599Z 
2022-11-23T01:19:07.7167833Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7168066Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7168304Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7168544Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7168794Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7169043Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7169270Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7169503Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7169817Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7170057Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7170308Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7170558Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7170794Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7171029Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7171290Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7171526Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7171772Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7172019Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7172251Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7172515Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7172749Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7172982Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7173231Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7173466Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7173699Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7173930Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7174172Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7174413Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7174660Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7175022Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7175261Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7175497Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7175724Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7175959Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7176212Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7176465Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7176701Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7176938Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7177176Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7177412Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7177656Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7177892Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7178125Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7178360Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7178597Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7178878Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7179128Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7179376Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7179609Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7179842Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7180066Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7180336Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7180587Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7180837Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7181072Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7181305Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7181542Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7181776Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7182010Z STAGE:2022-11-23 01:11:06 5846:5846 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7182258Z STAGE:2022-11-23 01:11:06 5845:5845 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7182324Z ok (2.370s)
2022-11-23T01:19:07.7182330Z 
2022-11-23T01:19:07.7182524Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7182600Z Ran 1 test in 2.371s
2022-11-23T01:19:07.7182606Z 
2022-11-23T01:19:07.7182663Z OK
2022-11-23T01:19:07.7182669Z 
2022-11-23T01:19:07.7182751Z Generating XML reports...
2022-11-23T01:19:07.7183095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011104.xml
2022-11-23T01:19:07.7183329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7183591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7183715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7183997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7184132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7184138Z 
2022-11-23T01:19:07.7184209Z Running tests...
2022-11-23T01:19:07.7184405Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7184636Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5905
2022-11-23T01:19:07.7184792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5906
2022-11-23T01:19:07.7184931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5907
2022-11-23T01:19:07.7185203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7185327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7185606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7185739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7185941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7186215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7186337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7186617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7186738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7186914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7187213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7187333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7187611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7187744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7187917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7188218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7188515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7188797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7188963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7189126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7189293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7189387Z skip: CUDA is not available. (2.077s)
2022-11-23T01:19:07.7189393Z 
2022-11-23T01:19:07.7189589Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7189663Z Ran 1 test in 2.077s
2022-11-23T01:19:07.7189669Z 
2022-11-23T01:19:07.7189738Z OK (skipped=1)
2022-11-23T01:19:07.7189743Z 
2022-11-23T01:19:07.7189822Z Generating XML reports...
2022-11-23T01:19:07.7190148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011108.xml
2022-11-23T01:19:07.7190381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7190652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7190773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7191055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7191189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7191194Z 
2022-11-23T01:19:07.7191265Z Running tests...
2022-11-23T01:19:07.7191458Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7192145Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7192154Z 
2022-11-23T01:19:07.7192345Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7192407Z Ran 1 test in 0.254s
2022-11-23T01:19:07.7192413Z 
2022-11-23T01:19:07.7192511Z OK (skipped=1)
2022-11-23T01:19:07.7192518Z 
2022-11-23T01:19:07.7192604Z Generating XML reports...
2022-11-23T01:19:07.7192944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011112.xml
2022-11-23T01:19:07.7193177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7193451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7193575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7193888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7194010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7194028Z 
2022-11-23T01:19:07.7194086Z Running tests...
2022-11-23T01:19:07.7194282Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7194661Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2022-11-23T01:19:07.7194667Z 
2022-11-23T01:19:07.7194857Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7194931Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7194937Z 
2022-11-23T01:19:07.7195005Z OK (skipped=1)
2022-11-23T01:19:07.7195010Z 
2022-11-23T01:19:07.7195091Z Generating XML reports...
2022-11-23T01:19:07.7195430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011114.xml
2022-11-23T01:19:07.7195648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7195919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7196044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7196324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7196457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7196462Z 
2022-11-23T01:19:07.7196533Z Running tests...
2022-11-23T01:19:07.7196728Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7197107Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2022-11-23T01:19:07.7197116Z 
2022-11-23T01:19:07.7197305Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7197367Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7197383Z 
2022-11-23T01:19:07.7197440Z OK (skipped=1)
2022-11-23T01:19:07.7197445Z 
2022-11-23T01:19:07.7197529Z Generating XML reports...
2022-11-23T01:19:07.7197866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011115.xml
2022-11-23T01:19:07.7198097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7198368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7198491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7198771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7198907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7198912Z 
2022-11-23T01:19:07.7198971Z Running tests...
2022-11-23T01:19:07.7199161Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7199423Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5988
2022-11-23T01:19:07.7199580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5989
2022-11-23T01:19:07.7199732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5990
2022-11-23T01:19:07.7200003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7200127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7200439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7200560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7200737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7201011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7201132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7201410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7201543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7201718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7201987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7202112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7202378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7202512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7202689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7202990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7203285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7203578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7203743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7203904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7204066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7204159Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:19:07.7204164Z 
2022-11-23T01:19:07.7204357Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7204431Z Ran 1 test in 2.173s
2022-11-23T01:19:07.7204437Z 
2022-11-23T01:19:07.7204508Z OK (skipped=1)
2022-11-23T01:19:07.7204513Z 
2022-11-23T01:19:07.7204595Z Generating XML reports...
2022-11-23T01:19:07.7204934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011117.xml
2022-11-23T01:19:07.7205162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7205440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7205562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7205829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7205991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7205996Z 
2022-11-23T01:19:07.7206068Z Running tests...
2022-11-23T01:19:07.7206262Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7206503Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6041
2022-11-23T01:19:07.7206657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6042
2022-11-23T01:19:07.7206805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6043
2022-11-23T01:19:07.7207114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7207224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7207503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7207636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7207814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7208082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7208204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7208481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7208615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7208789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7209046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7209172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7209454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7209587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7209831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7210131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7210428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7210717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7210882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7211033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7211195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7211298Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:19:07.7211304Z 
2022-11-23T01:19:07.7211498Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7211569Z Ran 1 test in 2.177s
2022-11-23T01:19:07.7211575Z 
2022-11-23T01:19:07.7211647Z OK (skipped=1)
2022-11-23T01:19:07.7211654Z 
2022-11-23T01:19:07.7211736Z Generating XML reports...
2022-11-23T01:19:07.7212076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011121.xml
2022-11-23T01:19:07.7212294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7212603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7212727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7213007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7213140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7213145Z 
2022-11-23T01:19:07.7213218Z Running tests...
2022-11-23T01:19:07.7213413Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7214135Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7214141Z 
2022-11-23T01:19:07.7214336Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7214411Z Ran 1 test in 0.254s
2022-11-23T01:19:07.7214416Z 
2022-11-23T01:19:07.7214474Z OK (skipped=1)
2022-11-23T01:19:07.7214490Z 
2022-11-23T01:19:07.7214560Z Generating XML reports...
2022-11-23T01:19:07.7214985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011125.xml
2022-11-23T01:19:07.7215219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7215496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7215619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7215900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7216037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7216043Z 
2022-11-23T01:19:07.7216114Z Running tests...
2022-11-23T01:19:07.7216295Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7217000Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:19:07.7217020Z 
2022-11-23T01:19:07.7217201Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7217276Z Ran 1 test in 0.253s
2022-11-23T01:19:07.7217281Z 
2022-11-23T01:19:07.7217355Z OK (skipped=1)
2022-11-23T01:19:07.7217360Z 
2022-11-23T01:19:07.7217441Z Generating XML reports...
2022-11-23T01:19:07.7217778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011127.xml
2022-11-23T01:19:07.7218010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7218284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7218407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7218676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7218812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7218818Z 
2022-11-23T01:19:07.7218887Z Running tests...
2022-11-23T01:19:07.7219079Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7219470Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s)
2022-11-23T01:19:07.7219476Z 
2022-11-23T01:19:07.7219668Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7219741Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7219746Z 
2022-11-23T01:19:07.7219816Z OK (skipped=1)
2022-11-23T01:19:07.7219820Z 
2022-11-23T01:19:07.7219905Z Generating XML reports...
2022-11-23T01:19:07.7220230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011129.xml
2022-11-23T01:19:07.7220495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7220770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7220891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7221171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7221305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7221310Z 
2022-11-23T01:19:07.7221380Z Running tests...
2022-11-23T01:19:07.7221571Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7221824Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6124
2022-11-23T01:19:07.7221978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6125
2022-11-23T01:19:07.7222133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6126
2022-11-23T01:19:07.7222406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7222528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7222810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7222943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7223120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7223391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7223501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7223782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7223915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7224093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7224365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7224487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7224764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7224897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7225073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7225359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7225659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7225947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7226138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7226302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7226461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7226561Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:19:07.7226567Z 
2022-11-23T01:19:07.7226760Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7226834Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7226865Z 
2022-11-23T01:19:07.7226923Z OK (skipped=1)
2022-11-23T01:19:07.7226928Z 
2022-11-23T01:19:07.7227014Z Generating XML reports...
2022-11-23T01:19:07.7227356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011130.xml
2022-11-23T01:19:07.7227587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7227860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7227985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7228266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7228400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7228406Z 
2022-11-23T01:19:07.7228464Z Running tests...
2022-11-23T01:19:07.7228662Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7228896Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6177
2022-11-23T01:19:07.7229049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6178
2022-11-23T01:19:07.7229203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6179
2022-11-23T01:19:07.7229472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7229592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7229872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7230006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7230169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7230468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7230740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7230866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7231146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7231278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7231452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7231717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7231839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7232105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7232241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7232416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7232737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7233037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7233200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7233361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7233522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7233673Z skip: Need at least 3 CUDA devices (2.178s)
2022-11-23T01:19:07.7233691Z 
2022-11-23T01:19:07.7233874Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7233949Z Ran 1 test in 2.178s
2022-11-23T01:19:07.7233954Z 
2022-11-23T01:19:07.7234028Z OK (skipped=1)
2022-11-23T01:19:07.7234033Z 
2022-11-23T01:19:07.7234118Z Generating XML reports...
2022-11-23T01:19:07.7234459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011134.xml
2022-11-23T01:19:07.7234690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7234962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7235083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7235350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7235486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7235492Z 
2022-11-23T01:19:07.7235560Z Running tests...
2022-11-23T01:19:07.7235756Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7236109Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.004s)
2022-11-23T01:19:07.7236115Z 
2022-11-23T01:19:07.7236306Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7236380Z Ran 1 test in 0.004s
2022-11-23T01:19:07.7236385Z 
2022-11-23T01:19:07.7236456Z OK (skipped=1)
2022-11-23T01:19:07.7236460Z 
2022-11-23T01:19:07.7236543Z Generating XML reports...
2022-11-23T01:19:07.7236874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011138.xml
2022-11-23T01:19:07.7237106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7237376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7237498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7237779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7237912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7237917Z 
2022-11-23T01:19:07.7237987Z Running tests...
2022-11-23T01:19:07.7238177Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7238521Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s)
2022-11-23T01:19:07.7238528Z 
2022-11-23T01:19:07.7238704Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7238775Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7238780Z 
2022-11-23T01:19:07.7238850Z OK (skipped=1)
2022-11-23T01:19:07.7238855Z 
2022-11-23T01:19:07.7238938Z Generating XML reports...
2022-11-23T01:19:07.7239305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011140.xml
2022-11-23T01:19:07.7239539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7239810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7239931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7240209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7240359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7240364Z 
2022-11-23T01:19:07.7240434Z Running tests...
2022-11-23T01:19:07.7240626Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7240854Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6250
2022-11-23T01:19:07.7241010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6251
2022-11-23T01:19:07.7241162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6252
2022-11-23T01:19:07.7241435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7241556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7241826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7241963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7242138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7242408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7242531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7242808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7242939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7243116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7243383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7243495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7243776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7243907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7244083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7244381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7244676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7244966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7245128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7245291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7245441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7245627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt38h8ta7
2022-11-23T01:19:07.7245844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt38h8ta7/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7246028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiaqza8i
2022-11-23T01:19:07.7246206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qbi6247
2022-11-23T01:19:07.7246398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiaqza8i/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7246583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qbi6247/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7247297Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7247981Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7248654Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7249560Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:19:07.7249809Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:19:07.7249980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7250891Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:19:07.7251063Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:19:07.7251233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7252130Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:19:07.7252291Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:19:07.7252490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7253175Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7253859Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7254573Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7255334Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7256004Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7256675Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7257341Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7258007Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7258668Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7259333Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7260048Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7260706Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7261408Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7262061Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7262714Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:19:07.7262782Z ok (2.076s)
2022-11-23T01:19:07.7262788Z 
2022-11-23T01:19:07.7262976Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7263051Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7263056Z 
2022-11-23T01:19:07.7263115Z OK
2022-11-23T01:19:07.7263120Z 
2022-11-23T01:19:07.7263203Z Generating XML reports...
2022-11-23T01:19:07.7263545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011141.xml
2022-11-23T01:19:07.7263777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7264049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7264174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7264442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7264578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7264584Z 
2022-11-23T01:19:07.7264655Z Running tests...
2022-11-23T01:19:07.7264848Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7265154Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.004s)
2022-11-23T01:19:07.7265160Z 
2022-11-23T01:19:07.7265351Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7265425Z Ran 1 test in 0.004s
2022-11-23T01:19:07.7265430Z 
2022-11-23T01:19:07.7265498Z OK (skipped=1)
2022-11-23T01:19:07.7265503Z 
2022-11-23T01:19:07.7265587Z Generating XML reports...
2022-11-23T01:19:07.7265916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011145.xml
2022-11-23T01:19:07.7266146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7266454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7266578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7266856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7266989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7266995Z 
2022-11-23T01:19:07.7267066Z Running tests...
2022-11-23T01:19:07.7267258Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7267581Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s)
2022-11-23T01:19:07.7267613Z 
2022-11-23T01:19:07.7267790Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7267862Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7267868Z 
2022-11-23T01:19:07.7267941Z OK (skipped=1)
2022-11-23T01:19:07.7267948Z 
2022-11-23T01:19:07.7268030Z Generating XML reports...
2022-11-23T01:19:07.7268367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011147.xml
2022-11-23T01:19:07.7268597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7268867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7268989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7269270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7269392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7269397Z 
2022-11-23T01:19:07.7269463Z Running tests...
2022-11-23T01:19:07.7269654Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7270346Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2022-11-23T01:19:07.7270353Z 
2022-11-23T01:19:07.7270546Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7270618Z Ran 1 test in 0.256s
2022-11-23T01:19:07.7270625Z 
2022-11-23T01:19:07.7270694Z OK (skipped=1)
2022-11-23T01:19:07.7270699Z 
2022-11-23T01:19:07.7270780Z Generating XML reports...
2022-11-23T01:19:07.7271118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011148.xml
2022-11-23T01:19:07.7271337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7271610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7271731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7272011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7272145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7272150Z 
2022-11-23T01:19:07.7272220Z Running tests...
2022-11-23T01:19:07.7272415Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7273134Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7273142Z 
2022-11-23T01:19:07.7273337Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7273410Z Ran 1 test in 0.254s
2022-11-23T01:19:07.7273415Z 
2022-11-23T01:19:07.7273471Z OK (skipped=1)
2022-11-23T01:19:07.7273485Z 
2022-11-23T01:19:07.7273555Z Generating XML reports...
2022-11-23T01:19:07.7273895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011150.xml
2022-11-23T01:19:07.7274127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7274429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7274552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7274833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7274966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7274971Z 
2022-11-23T01:19:07.7275041Z Running tests...
2022-11-23T01:19:07.7275222Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7275476Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6343
2022-11-23T01:19:07.7275633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6344
2022-11-23T01:19:07.7275786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6345
2022-11-23T01:19:07.7276060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7276183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7276468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7276602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7276779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7277040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7277163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7277442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7277576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7277753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7278027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7278149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7278429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7278550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7278723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7279020Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7279324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7279612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7279802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7279968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7280128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7280234Z skip: Need at least 3 CUDA devices (2.171s)
2022-11-23T01:19:07.7280239Z 
2022-11-23T01:19:07.7280422Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7280495Z Ran 1 test in 2.171s
2022-11-23T01:19:07.7280500Z 
2022-11-23T01:19:07.7280596Z OK (skipped=1)
2022-11-23T01:19:07.7280602Z 
2022-11-23T01:19:07.7280687Z Generating XML reports...
2022-11-23T01:19:07.7281027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011152.xml
2022-11-23T01:19:07.7281257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7281529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7281652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7281929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7282052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7282057Z 
2022-11-23T01:19:07.7282126Z Running tests...
2022-11-23T01:19:07.7282319Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7282562Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6396
2022-11-23T01:19:07.7282715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6397
2022-11-23T01:19:07.7282865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6398
2022-11-23T01:19:07.7283136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7283256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7283519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7283652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7283825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7284100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7284221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7284500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7284637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7284813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7285083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7285192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7285467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7285601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7285776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7286075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7286400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7286695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7286861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7287023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7287172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7287301Z skip: Need at least 3 CUDA devices (2.176s)
2022-11-23T01:19:07.7287308Z 
2022-11-23T01:19:07.7287503Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7287577Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7287583Z 
2022-11-23T01:19:07.7287651Z OK (skipped=1)
2022-11-23T01:19:07.7287656Z 
2022-11-23T01:19:07.7287738Z Generating XML reports...
2022-11-23T01:19:07.7288078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011156.xml
2022-11-23T01:19:07.7288309Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7288579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7288689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7288971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7289108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7289114Z 
2022-11-23T01:19:07.7289186Z Running tests...
2022-11-23T01:19:07.7289380Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7290147Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:19:07.7290156Z 
2022-11-23T01:19:07.7290349Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7290419Z Ran 1 test in 0.253s
2022-11-23T01:19:07.7290425Z 
2022-11-23T01:19:07.7290494Z OK (skipped=1)
2022-11-23T01:19:07.7290501Z 
2022-11-23T01:19:07.7290571Z Generating XML reports...
2022-11-23T01:19:07.7290909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011200.xml
2022-11-23T01:19:07.7291139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7291413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7291534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7291813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7291946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7291952Z 
2022-11-23T01:19:07.7292022Z Running tests...
2022-11-23T01:19:07.7292216Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7292445Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6459
2022-11-23T01:19:07.7292598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6460
2022-11-23T01:19:07.7292751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6461
2022-11-23T01:19:07.7293061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7293185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7293468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7293603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7293780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7294079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7294189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7294468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7294605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7294779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7295161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7295284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7295566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7295700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7295866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7296167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7296467Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7296758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7296923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7297086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7297248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7297354Z skip: Need at least 3 CUDA devices (2.175s)
2022-11-23T01:19:07.7297360Z 
2022-11-23T01:19:07.7297555Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7297617Z Ran 1 test in 2.175s
2022-11-23T01:19:07.7297635Z 
2022-11-23T01:19:07.7297692Z OK (skipped=1)
2022-11-23T01:19:07.7297698Z 
2022-11-23T01:19:07.7297779Z Generating XML reports...
2022-11-23T01:19:07.7298124Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011202.xml
2022-11-23T01:19:07.7298356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7298628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7298749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7299031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7299167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7299172Z 
2022-11-23T01:19:07.7299230Z Running tests...
2022-11-23T01:19:07.7299422Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7299738Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6512
2022-11-23T01:19:07.7299896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6513
2022-11-23T01:19:07.7300047Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6514
2022-11-23T01:19:07.7300319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7300440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7300719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7300898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7301061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7301332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7301453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7301731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7301864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7302037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7302335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7302607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7302719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7302998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7303129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7303306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7303602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7303896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7304058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7304222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7304382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7304472Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:19:07.7304489Z 
2022-11-23T01:19:07.7304674Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7304748Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7304753Z 
2022-11-23T01:19:07.7304824Z OK (skipped=1)
2022-11-23T01:19:07.7304830Z 
2022-11-23T01:19:07.7304911Z Generating XML reports...
2022-11-23T01:19:07.7305250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011206.xml
2022-11-23T01:19:07.7305482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7305757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7305878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7306145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7306312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7306318Z 
2022-11-23T01:19:07.7306388Z Running tests...
2022-11-23T01:19:07.7306583Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7306852Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6565
2022-11-23T01:19:07.7307007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6566
2022-11-23T01:19:07.7307159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6567
2022-11-23T01:19:07.7307459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7307581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7307848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7307981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7308156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7308428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7308548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7308826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7308961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7309136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7309396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7309520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7309803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7309936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7310110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7310406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7310704Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7310993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7311156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7311307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7311469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7311569Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:19:07.7311574Z 
2022-11-23T01:19:07.7311766Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7311839Z Ran 1 test in 2.174s
2022-11-23T01:19:07.7311844Z 
2022-11-23T01:19:07.7311915Z OK (skipped=1)
2022-11-23T01:19:07.7311922Z 
2022-11-23T01:19:07.7312005Z Generating XML reports...
2022-11-23T01:19:07.7312345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011210.xml
2022-11-23T01:19:07.7312577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7312868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7312991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7313269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7313402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7313407Z 
2022-11-23T01:19:07.7313477Z Running tests...
2022-11-23T01:19:07.7313671Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7314015Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6618
2022-11-23T01:19:07.7314170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6619
2022-11-23T01:19:07.7314323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6620
2022-11-23T01:19:07.7314585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7314707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7314986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7315120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7315297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7315568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7315689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7315971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7316090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7316265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7316537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7316659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7316939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7317074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7317250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7317546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7317839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7318118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7318282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7318444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7318605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7318711Z skip: Need at least 2 CUDA devices (2.074s)
2022-11-23T01:19:07.7318716Z 
2022-11-23T01:19:07.7318910Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7318985Z Ran 1 test in 2.075s
2022-11-23T01:19:07.7318991Z 
2022-11-23T01:19:07.7319063Z OK (skipped=1)
2022-11-23T01:19:07.7319068Z 
2022-11-23T01:19:07.7319185Z Generating XML reports...
2022-11-23T01:19:07.7319515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011213.xml
2022-11-23T01:19:07.7319744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7320015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7320137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7320414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7320572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7320577Z 
2022-11-23T01:19:07.7320647Z Running tests...
2022-11-23T01:19:07.7320838Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7321155Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6671
2022-11-23T01:19:07.7321298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6672
2022-11-23T01:19:07.7321449Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6673
2022-11-23T01:19:07.7321724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7321845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7322122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7322259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7322435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7322706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7322816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7323095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7323229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7323402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7323670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7323792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7324071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7324205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7324379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7324665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7324962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7325253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7325421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7325581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7325770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7325874Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:19:07.7325879Z 
2022-11-23T01:19:07.7326076Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7326150Z Ran 1 test in 2.174s
2022-11-23T01:19:07.7326155Z 
2022-11-23T01:19:07.7326212Z OK (skipped=1)
2022-11-23T01:19:07.7326217Z 
2022-11-23T01:19:07.7326300Z Generating XML reports...
2022-11-23T01:19:07.7326639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011217.xml
2022-11-23T01:19:07.7326905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7327177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7327301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7327583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7327718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7327723Z 
2022-11-23T01:19:07.7327793Z Running tests...
2022-11-23T01:19:07.7327975Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7328294Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6724
2022-11-23T01:19:07.7328451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6725
2022-11-23T01:19:07.7328602Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6726
2022-11-23T01:19:07.7328870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7329000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7329276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7329409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7329572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7329925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7330049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7330332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7330465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7330641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7330942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7331213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7331334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7331602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7331736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7331912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7332262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7332590Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7332757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7332918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7333077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7333180Z skip: Need at least 2 CUDA devices (2.077s)
2022-11-23T01:19:07.7333185Z 
2022-11-23T01:19:07.7333369Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7333475Z Ran 1 test in 2.077s
2022-11-23T01:19:07.7333480Z 
2022-11-23T01:19:07.7333553Z OK (skipped=1)
2022-11-23T01:19:07.7333558Z 
2022-11-23T01:19:07.7333641Z Generating XML reports...
2022-11-23T01:19:07.7333983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011221.xml
2022-11-23T01:19:07.7334215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7334489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7334610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7334997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7335121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7335126Z 
2022-11-23T01:19:07.7335199Z Running tests...
2022-11-23T01:19:07.7335393Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7335705Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6777
2022-11-23T01:19:07.7335860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6778
2022-11-23T01:19:07.7336012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6779
2022-11-23T01:19:07.7336283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7336405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7336685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7336809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7336982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7337253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7337378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7337656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7337790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7337964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7338235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7338345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7338626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7338758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7338932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7339306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7339605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7339894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7340060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7340257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7340408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7340510Z skip: Need at least 2 CUDA devices (2.072s)
2022-11-23T01:19:07.7340516Z 
2022-11-23T01:19:07.7340711Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7340787Z Ran 1 test in 2.072s
2022-11-23T01:19:07.7340792Z 
2022-11-23T01:19:07.7340862Z OK (skipped=1)
2022-11-23T01:19:07.7340867Z 
2022-11-23T01:19:07.7340952Z Generating XML reports...
2022-11-23T01:19:07.7341293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011225.xml
2022-11-23T01:19:07.7341526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7341797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7341909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7342189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7342324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7342329Z 
2022-11-23T01:19:07.7342402Z Running tests...
2022-11-23T01:19:07.7342593Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7342908Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6830
2022-11-23T01:19:07.7343063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6831
2022-11-23T01:19:07.7343214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6832
2022-11-23T01:19:07.7343489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7343600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7343882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7344018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7344199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7344470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7344591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7344869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7345004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7345169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7345441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7345591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7345870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7346002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7346176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7346474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7346771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7347103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7347254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7347416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7347576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7347680Z skip: Need at least 2 CUDA devices (2.172s)
2022-11-23T01:19:07.7347685Z 
2022-11-23T01:19:07.7347883Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7347957Z Ran 1 test in 2.172s
2022-11-23T01:19:07.7347963Z 
2022-11-23T01:19:07.7348032Z OK (skipped=1)
2022-11-23T01:19:07.7348037Z 
2022-11-23T01:19:07.7348119Z Generating XML reports...
2022-11-23T01:19:07.7348465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011229.xml
2022-11-23T01:19:07.7348685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7348960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7349086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7349369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7349502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7349507Z 
2022-11-23T01:19:07.7349577Z Running tests...
2022-11-23T01:19:07.7349771Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7350084Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6883
2022-11-23T01:19:07.7350242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6884
2022-11-23T01:19:07.7350380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6885
2022-11-23T01:19:07.7350653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7350776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7351054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7351187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7351361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7351632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7351754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7352021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7352181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7352357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7352627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7352749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7353025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7353158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7353367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7353666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7353954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7354248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7354415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7354578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7354739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7354844Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:19:07.7354850Z 
2022-11-23T01:19:07.7355044Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7355117Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7355122Z 
2022-11-23T01:19:07.7355192Z OK (skipped=1)
2022-11-23T01:19:07.7355197Z 
2022-11-23T01:19:07.7355267Z Generating XML reports...
2022-11-23T01:19:07.7355611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011233.xml
2022-11-23T01:19:07.7355842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7356114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7356237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7356515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7356650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7356656Z 
2022-11-23T01:19:07.7356728Z Running tests...
2022-11-23T01:19:07.7356910Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7357226Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6936
2022-11-23T01:19:07.7357386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6937
2022-11-23T01:19:07.7357539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6938
2022-11-23T01:19:07.7357811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7357933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7358216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7358349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7358524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7358813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7358935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7359217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7359349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7359524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7359793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7359943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7360224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7360360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7360525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7360823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7361116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7361405Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7361571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7361732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7361893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7361997Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:19:07.7362003Z 
2022-11-23T01:19:07.7362198Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7362259Z Ran 1 test in 2.075s
2022-11-23T01:19:07.7362265Z 
2022-11-23T01:19:07.7362333Z OK (skipped=1)
2022-11-23T01:19:07.7362338Z 
2022-11-23T01:19:07.7362420Z Generating XML reports...
2022-11-23T01:19:07.7362766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011237.xml
2022-11-23T01:19:07.7362999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7363275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7363397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7363678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7363813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7363818Z 
2022-11-23T01:19:07.7363876Z Running tests...
2022-11-23T01:19:07.7364072Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7364383Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6989
2022-11-23T01:19:07.7364537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6990
2022-11-23T01:19:07.7364691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6991
2022-11-23T01:19:07.7364962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7365082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7365389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7365523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7365687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7365956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7366073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7366389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7366524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7366699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7366977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7367099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7367379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7367512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7367687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7367973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7368273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7368563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7368730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7368890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7369052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7369157Z skip: Need at least 2 CUDA devices (2.172s)
2022-11-23T01:19:07.7369163Z 
2022-11-23T01:19:07.7369358Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7369431Z Ran 1 test in 2.172s
2022-11-23T01:19:07.7369438Z 
2022-11-23T01:19:07.7369497Z OK (skipped=1)
2022-11-23T01:19:07.7369501Z 
2022-11-23T01:19:07.7369584Z Generating XML reports...
2022-11-23T01:19:07.7370006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011240.xml
2022-11-23T01:19:07.7370243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7370514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7370639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7370921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7371055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7371060Z 
2022-11-23T01:19:07.7371131Z Running tests...
2022-11-23T01:19:07.7371312Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7371585Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7042
2022-11-23T01:19:07.7371739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7043
2022-11-23T01:19:07.7371924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7044
2022-11-23T01:19:07.7372199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7372321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7372600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7372733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7372897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7373198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7373318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7373598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7373735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7373910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7374205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7374474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7374597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7374949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7375087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7375262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7375562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7375853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7376016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7376177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7376340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7376444Z skip: Need at least 2 CUDA devices (2.175s)
2022-11-23T01:19:07.7376450Z 
2022-11-23T01:19:07.7376630Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7376704Z Ran 1 test in 2.175s
2022-11-23T01:19:07.7376710Z 
2022-11-23T01:19:07.7376778Z OK (skipped=1)
2022-11-23T01:19:07.7376785Z 
2022-11-23T01:19:07.7376867Z Generating XML reports...
2022-11-23T01:19:07.7377206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011244.xml
2022-11-23T01:19:07.7377437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7377711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7377833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7378114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7378237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7378243Z 
2022-11-23T01:19:07.7378314Z Running tests...
2022-11-23T01:19:07.7378509Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7378822Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7095
2022-11-23T01:19:07.7378978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7096
2022-11-23T01:19:07.7379128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7097
2022-11-23T01:19:07.7379400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7379554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7379819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7379953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7380128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7380400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7380521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7380800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7380933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7381108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7381379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7381489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7381767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7381931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7382105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7382400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7382697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7382986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7383151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7383312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7383464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7383566Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:19:07.7383572Z 
2022-11-23T01:19:07.7383767Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7383840Z Ran 1 test in 2.173s
2022-11-23T01:19:07.7383845Z 
2022-11-23T01:19:07.7383915Z OK (skipped=1)
2022-11-23T01:19:07.7383920Z 
2022-11-23T01:19:07.7384002Z Generating XML reports...
2022-11-23T01:19:07.7384344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011248.xml
2022-11-23T01:19:07.7384576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7384837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7384960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7385271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7385406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7385411Z 
2022-11-23T01:19:07.7385483Z Running tests...
2022-11-23T01:19:07.7385675Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7385997Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s)
2022-11-23T01:19:07.7386003Z 
2022-11-23T01:19:07.7386193Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7386294Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7386299Z 
2022-11-23T01:19:07.7386356Z OK (skipped=1)
2022-11-23T01:19:07.7386372Z 
2022-11-23T01:19:07.7386443Z Generating XML reports...
2022-11-23T01:19:07.7386780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011252.xml
2022-11-23T01:19:07.7387015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7387286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7387408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7387687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7387822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7387830Z 
2022-11-23T01:19:07.7387900Z Running tests...
2022-11-23T01:19:07.7388079Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7388305Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7158
2022-11-23T01:19:07.7388462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7159
2022-11-23T01:19:07.7388616Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7160
2022-11-23T01:19:07.7388883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7389004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7389279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7389410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7389585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7389846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7389965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7390246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7390380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7390555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7390852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7391120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7391243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7391509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7391640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7391841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7392139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7392432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7392595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7392754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7392955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7393056Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:19:07.7393063Z 
2022-11-23T01:19:07.7393243Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7393317Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7393324Z 
2022-11-23T01:19:07.7393393Z OK (skipped=1)
2022-11-23T01:19:07.7393398Z 
2022-11-23T01:19:07.7393480Z Generating XML reports...
2022-11-23T01:19:07.7393821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011254.xml
2022-11-23T01:19:07.7394052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7394326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7394449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7394731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7394852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7394858Z 
2022-11-23T01:19:07.7394929Z Running tests...
2022-11-23T01:19:07.7395122Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7395363Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7211
2022-11-23T01:19:07.7395518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7212
2022-11-23T01:19:07.7395669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7213
2022-11-23T01:19:07.7395939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7396061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7396330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7396463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7396639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7396913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7397035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7397315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7397447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7397621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7397894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7398002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7398304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7398440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7398614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7398914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7399207Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7399500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7399690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7399851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7400002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7400102Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:19:07.7400108Z 
2022-11-23T01:19:07.7400301Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7400375Z Ran 1 test in 2.075s
2022-11-23T01:19:07.7400380Z 
2022-11-23T01:19:07.7400450Z OK (skipped=1)
2022-11-23T01:19:07.7400455Z 
2022-11-23T01:19:07.7400537Z Generating XML reports...
2022-11-23T01:19:07.7400880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011258.xml
2022-11-23T01:19:07.7401112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7401384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7401494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7401776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7401909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7401914Z 
2022-11-23T01:19:07.7401985Z Running tests...
2022-11-23T01:19:07.7402176Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7402411Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7264
2022-11-23T01:19:07.7402563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7265
2022-11-23T01:19:07.7402717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7266
2022-11-23T01:19:07.7402977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7403098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7403378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7403509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7403685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7403979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7404248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7404369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7404648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7404767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7404965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7405237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7405357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7405635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7405767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7405971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7406267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7406562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7406713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7406875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7407062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe44hirz
2022-11-23T01:19:07.7407246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9s8m9yw
2022-11-23T01:19:07.7407436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe44hirz/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7407626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9s8m9yw/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7407787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7407969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xu4rr09
2022-11-23T01:19:07.7408150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xu4rr09/_remote_module_non_scriptable.py
2022-11-23T01:19:07.7408318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7408485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7408652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:19:07.7408718Z ok (2.377s)
2022-11-23T01:19:07.7408724Z 
2022-11-23T01:19:07.7408918Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7408993Z Ran 1 test in 2.378s
2022-11-23T01:19:07.7408999Z 
2022-11-23T01:19:07.7409057Z OK
2022-11-23T01:19:07.7409062Z 
2022-11-23T01:19:07.7409144Z Generating XML reports...
2022-11-23T01:19:07.7409474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011301.xml
2022-11-23T01:19:07.7409795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7410073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7410195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7410476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7410609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7410615Z 
2022-11-23T01:19:07.7410688Z Running tests...
2022-11-23T01:19:07.7410882Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7411104Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7326
2022-11-23T01:19:07.7411256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7327
2022-11-23T01:19:07.7411440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7328
2022-11-23T01:19:07.7411712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7411834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7412111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7412243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7412447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7412716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7412825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7413109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7413246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7413423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7413695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7413814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7414092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7414226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7414389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7414690Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7415070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7415363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7415527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7415689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7415852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7415946Z skip: CUDA is not available. (2.171s)
2022-11-23T01:19:07.7415952Z 
2022-11-23T01:19:07.7416148Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7416210Z Ran 1 test in 2.171s
2022-11-23T01:19:07.7416229Z 
2022-11-23T01:19:07.7416290Z OK (skipped=1)
2022-11-23T01:19:07.7416296Z 
2022-11-23T01:19:07.7416381Z Generating XML reports...
2022-11-23T01:19:07.7416718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011305.xml
2022-11-23T01:19:07.7416951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7417225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7417348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7417632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7417767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7417772Z 
2022-11-23T01:19:07.7417831Z Running tests...
2022-11-23T01:19:07.7418067Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7418424Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2022-11-23T01:19:07.7418430Z 
2022-11-23T01:19:07.7418622Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7418697Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7418703Z 
2022-11-23T01:19:07.7418772Z OK (skipped=1)
2022-11-23T01:19:07.7418777Z 
2022-11-23T01:19:07.7418858Z Generating XML reports...
2022-11-23T01:19:07.7419201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011309.xml
2022-11-23T01:19:07.7419464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7419724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7419848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7420127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7420265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7420270Z 
2022-11-23T01:19:07.7420340Z Running tests...
2022-11-23T01:19:07.7420531Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7420870Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2022-11-23T01:19:07.7420878Z 
2022-11-23T01:19:07.7421067Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7421140Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7421146Z 
2022-11-23T01:19:07.7421202Z OK (skipped=1)
2022-11-23T01:19:07.7421207Z 
2022-11-23T01:19:07.7421289Z Generating XML reports...
2022-11-23T01:19:07.7421630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011311.xml
2022-11-23T01:19:07.7421863Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7422134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7422254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7422532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7422667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7422672Z 
2022-11-23T01:19:07.7422742Z Running tests...
2022-11-23T01:19:07.7422922Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7423292Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2022-11-23T01:19:07.7423298Z 
2022-11-23T01:19:07.7423487Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7423565Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7423570Z 
2022-11-23T01:19:07.7423641Z OK (skipped=1)
2022-11-23T01:19:07.7423646Z 
2022-11-23T01:19:07.7423728Z Generating XML reports...
2022-11-23T01:19:07.7424067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011313.xml
2022-11-23T01:19:07.7424300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7424575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7424685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7424994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7425129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7425135Z 
2022-11-23T01:19:07.7425205Z Running tests...
2022-11-23T01:19:07.7425399Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7425748Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:19:07.7425754Z 
2022-11-23T01:19:07.7425945Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7426048Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7426053Z 
2022-11-23T01:19:07.7426110Z OK (skipped=1)
2022-11-23T01:19:07.7426127Z 
2022-11-23T01:19:07.7426196Z Generating XML reports...
2022-11-23T01:19:07.7426537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011314.xml
2022-11-23T01:19:07.7426767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7427040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7427162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7427441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7427576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7427583Z 
2022-11-23T01:19:07.7427653Z Running tests...
2022-11-23T01:19:07.7427831Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7428144Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s)
2022-11-23T01:19:07.7428150Z 
2022-11-23T01:19:07.7428342Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7428415Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7428421Z 
2022-11-23T01:19:07.7428490Z OK (skipped=1)
2022-11-23T01:19:07.7428495Z 
2022-11-23T01:19:07.7428577Z Generating XML reports...
2022-11-23T01:19:07.7428911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011316.xml
2022-11-23T01:19:07.7429141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7429412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7429523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7429802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7429936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7429942Z 
2022-11-23T01:19:07.7430012Z Running tests...
2022-11-23T01:19:07.7430206Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7430435Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7429
2022-11-23T01:19:07.7430589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7430
2022-11-23T01:19:07.7430740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7431
2022-11-23T01:19:07.7431011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7431121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7431400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7431570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7431747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7432020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7432141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7432421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7432556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7432750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7433020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7433141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7433422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7433555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7433731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7434034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7434333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7434623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7434778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7434944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7435105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7435211Z skip: Need at least 2 CUDA devices (2.175s)
2022-11-23T01:19:07.7435217Z 
2022-11-23T01:19:07.7435410Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7435484Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7435492Z 
2022-11-23T01:19:07.7435560Z OK (skipped=1)
2022-11-23T01:19:07.7435564Z 
2022-11-23T01:19:07.7435647Z Generating XML reports...
2022-11-23T01:19:07.7435986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011317.xml
2022-11-23T01:19:07.7436206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7436480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7436603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7436884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7437017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7437023Z 
2022-11-23T01:19:07.7437092Z Running tests...
2022-11-23T01:19:07.7437284Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7437997Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.265s)
2022-11-23T01:19:07.7438006Z 
2022-11-23T01:19:07.7438226Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7438302Z Ran 1 test in 0.265s
2022-11-23T01:19:07.7438307Z 
2022-11-23T01:19:07.7438365Z OK (skipped=1)
2022-11-23T01:19:07.7438371Z 
2022-11-23T01:19:07.7438452Z Generating XML reports...
2022-11-23T01:19:07.7438791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011321.xml
2022-11-23T01:19:07.7439024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7439298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7439496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7439777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7439908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7439914Z 
2022-11-23T01:19:07.7439975Z Running tests...
2022-11-23T01:19:07.7440166Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7440502Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:19:07.7440508Z 
2022-11-23T01:19:07.7440697Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7440770Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7440775Z 
2022-11-23T01:19:07.7440844Z OK (skipped=1)
2022-11-23T01:19:07.7440851Z 
2022-11-23T01:19:07.7440933Z Generating XML reports...
2022-11-23T01:19:07.7441271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011323.xml
2022-11-23T01:19:07.7441502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7441764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7441885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7442165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7442299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7442304Z 
2022-11-23T01:19:07.7442373Z Running tests...
2022-11-23T01:19:07.7442565Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7442903Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.002s)
2022-11-23T01:19:07.7442909Z 
2022-11-23T01:19:07.7443098Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7443170Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7443175Z 
2022-11-23T01:19:07.7443233Z OK (skipped=1)
2022-11-23T01:19:07.7443251Z 
2022-11-23T01:19:07.7443322Z Generating XML reports...
2022-11-23T01:19:07.7443658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011325.xml
2022-11-23T01:19:07.7443892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7444163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7444285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7444568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7444702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7444708Z 
2022-11-23T01:19:07.7444778Z Running tests...
2022-11-23T01:19:07.7444958Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7445222Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7512
2022-11-23T01:19:07.7445380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7513
2022-11-23T01:19:07.7445532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7514
2022-11-23T01:19:07.7445804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7445924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7446230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7446364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7446527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7446801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7446923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7447204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7447338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7447514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7447786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7447907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7448184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7448307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7448483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7448783Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7449077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7449368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7449536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7449760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7449924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7450028Z skip: Need at least 2 CUDA devices (2.175s)
2022-11-23T01:19:07.7450034Z 
2022-11-23T01:19:07.7450217Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7450291Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7450297Z 
2022-11-23T01:19:07.7450365Z OK (skipped=1)
2022-11-23T01:19:07.7450370Z 
2022-11-23T01:19:07.7450452Z Generating XML reports...
2022-11-23T01:19:07.7450795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011326.xml
2022-11-23T01:19:07.7451026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7451302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7451425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7451738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7451862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7451867Z 
2022-11-23T01:19:07.7451936Z Running tests...
2022-11-23T01:19:07.7452130Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7452829Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2022-11-23T01:19:07.7452863Z 
2022-11-23T01:19:07.7453057Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7453132Z Ran 1 test in 0.258s
2022-11-23T01:19:07.7453138Z 
2022-11-23T01:19:07.7453207Z OK (skipped=1)
2022-11-23T01:19:07.7453212Z 
2022-11-23T01:19:07.7453296Z Generating XML reports...
2022-11-23T01:19:07.7453637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011330.xml
2022-11-23T01:19:07.7453855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7454127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7454248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7454526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7454662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7454668Z 
2022-11-23T01:19:07.7454738Z Running tests...
2022-11-23T01:19:07.7455019Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7455361Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2022-11-23T01:19:07.7455367Z 
2022-11-23T01:19:07.7455555Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7455616Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7455633Z 
2022-11-23T01:19:07.7455690Z OK (skipped=1)
2022-11-23T01:19:07.7455694Z 
2022-11-23T01:19:07.7455776Z Generating XML reports...
2022-11-23T01:19:07.7456110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011332.xml
2022-11-23T01:19:07.7456342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7456618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7456739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7457019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7457151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7457156Z 
2022-11-23T01:19:07.7457214Z Running tests...
2022-11-23T01:19:07.7457406Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7458093Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:19:07.7458101Z 
2022-11-23T01:19:07.7458292Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7458365Z Ran 1 test in 0.255s
2022-11-23T01:19:07.7458414Z 
2022-11-23T01:19:07.7458485Z OK (skipped=1)
2022-11-23T01:19:07.7458490Z 
2022-11-23T01:19:07.7458572Z Generating XML reports...
2022-11-23T01:19:07.7458908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011334.xml
2022-11-23T01:19:07.7459137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7459408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7459518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7459832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7459964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7459970Z 
2022-11-23T01:19:07.7460041Z Running tests...
2022-11-23T01:19:07.7460239Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7460477Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7595
2022-11-23T01:19:07.7460634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7596
2022-11-23T01:19:07.7460788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7597
2022-11-23T01:19:07.7461057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7461169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7461443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7461577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7461761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7462033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7462155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7462433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7462566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7462733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7463033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7463301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7463422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7463703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7463836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7464009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7464307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7464605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7464760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7464921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7465113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7465220Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:19:07.7465225Z 
2022-11-23T01:19:07.7465422Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7465495Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7465501Z 
2022-11-23T01:19:07.7465569Z OK (skipped=1)
2022-11-23T01:19:07.7465575Z 
2022-11-23T01:19:07.7465656Z Generating XML reports...
2022-11-23T01:19:07.7465995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011336.xml
2022-11-23T01:19:07.7466243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7466513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7466634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7466917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7467051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7467057Z 
2022-11-23T01:19:07.7467127Z Running tests...
2022-11-23T01:19:07.7467318Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7467552Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7648
2022-11-23T01:19:07.7467694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7649
2022-11-23T01:19:07.7467846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7650
2022-11-23T01:19:07.7468116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7468238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7468520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7468654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7468830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7469098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7469218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7469487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7469620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7469795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7470067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7470189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7470467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7470600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7470774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7471073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7471359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7471678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7471841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7472004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7472167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7472271Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:19:07.7472276Z 
2022-11-23T01:19:07.7472471Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7472544Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7472577Z 
2022-11-23T01:19:07.7472636Z OK (skipped=1)
2022-11-23T01:19:07.7472653Z 
2022-11-23T01:19:07.7472723Z Generating XML reports...
2022-11-23T01:19:07.7473063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011340.xml
2022-11-23T01:19:07.7473299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7473573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7473694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7473972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7474106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7474112Z 
2022-11-23T01:19:07.7474182Z Running tests...
2022-11-23T01:19:07.7474366Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7474603Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7701
2022-11-23T01:19:07.7474757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7702
2022-11-23T01:19:07.7474911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7703
2022-11-23T01:19:07.7475182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7475303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7475584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7475719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7475896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7476157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7476278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7476559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7476693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7476868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7477140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7477260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7477538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7477660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7477832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7478159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7478453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7478745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7478909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7479072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7479233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7479367Z skip: Need at least 2 CUDA devices (2.172s)
2022-11-23T01:19:07.7479373Z 
2022-11-23T01:19:07.7479558Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7479631Z Ran 1 test in 2.172s
2022-11-23T01:19:07.7479636Z 
2022-11-23T01:19:07.7479705Z OK (skipped=1)
2022-11-23T01:19:07.7479712Z 
2022-11-23T01:19:07.7479795Z Generating XML reports...
2022-11-23T01:19:07.7480132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011344.xml
2022-11-23T01:19:07.7480363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7480637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7480757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7481040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7481163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7481168Z 
2022-11-23T01:19:07.7481239Z Running tests...
2022-11-23T01:19:07.7481431Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7482131Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2022-11-23T01:19:07.7482138Z 
2022-11-23T01:19:07.7482329Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7482404Z Ran 1 test in 0.259s
2022-11-23T01:19:07.7482411Z 
2022-11-23T01:19:07.7482480Z OK (skipped=1)
2022-11-23T01:19:07.7482485Z 
2022-11-23T01:19:07.7482569Z Generating XML reports...
2022-11-23T01:19:07.7482903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011348.xml
2022-11-23T01:19:07.7483133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7483395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7483517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7483796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7483931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7483937Z 
2022-11-23T01:19:07.7484007Z Running tests...
2022-11-23T01:19:07.7484200Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7484904Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.258s)
2022-11-23T01:19:07.7484910Z 
2022-11-23T01:19:07.7485102Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7485174Z Ran 1 test in 0.258s
2022-11-23T01:19:07.7485180Z 
2022-11-23T01:19:07.7485249Z OK (skipped=1)
2022-11-23T01:19:07.7485255Z 
2022-11-23T01:19:07.7485325Z Generating XML reports...
2022-11-23T01:19:07.7485662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011349.xml
2022-11-23T01:19:07.7485892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7486216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7486338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7486620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7486753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7486758Z 
2022-11-23T01:19:07.7486826Z Running tests...
2022-11-23T01:19:07.7487007Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7487716Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:19:07.7487736Z 
2022-11-23T01:19:07.7487913Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7487987Z Ran 1 test in 0.256s
2022-11-23T01:19:07.7487992Z 
2022-11-23T01:19:07.7488061Z OK (skipped=1)
2022-11-23T01:19:07.7488066Z 
2022-11-23T01:19:07.7488150Z Generating XML reports...
2022-11-23T01:19:07.7488487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011351.xml
2022-11-23T01:19:07.7488716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7488989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7489112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7489381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7489522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7489527Z 
2022-11-23T01:19:07.7489598Z Running tests...
2022-11-23T01:19:07.7489863Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7490229Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.003s)
2022-11-23T01:19:07.7490235Z 
2022-11-23T01:19:07.7490426Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7490498Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7490503Z 
2022-11-23T01:19:07.7490573Z OK (skipped=1)
2022-11-23T01:19:07.7490578Z 
2022-11-23T01:19:07.7490659Z Generating XML reports...
2022-11-23T01:19:07.7490984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011353.xml
2022-11-23T01:19:07.7491220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7491488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7491611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7491922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7492058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7492063Z 
2022-11-23T01:19:07.7492133Z Running tests...
2022-11-23T01:19:07.7492326Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7492564Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7794
2022-11-23T01:19:07.7492741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7795
2022-11-23T01:19:07.7492894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7796
2022-11-23T01:19:07.7493167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7493288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7493570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7493701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7493878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7494146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7494266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7494533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7494665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7494841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7495227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7495498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7495618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7495895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7496028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7496193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7496489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7496787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7496949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7497111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7497269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7497372Z skip: Need at least 3 CUDA devices (2.175s)
2022-11-23T01:19:07.7497379Z 
2022-11-23T01:19:07.7497572Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7497647Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7497653Z 
2022-11-23T01:19:07.7497711Z OK (skipped=1)
2022-11-23T01:19:07.7497716Z 
2022-11-23T01:19:07.7497800Z Generating XML reports...
2022-11-23T01:19:07.7498141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011355.xml
2022-11-23T01:19:07.7498415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7498690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7498812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7499091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7499223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7499229Z 
2022-11-23T01:19:07.7499300Z Running tests...
2022-11-23T01:19:07.7499516Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7499746Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7847
2022-11-23T01:19:07.7499900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7848
2022-11-23T01:19:07.7500055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7849
2022-11-23T01:19:07.7500324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7500443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7500722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7500855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7501019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7501292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7501413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7501695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7501828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7502003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7502271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7502393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7502672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7502794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7502970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7503267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7503561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7503851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7504014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7504180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7504343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7504515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7504676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7505004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7505175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7505468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7505759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7505826Z ok (2.373s)
2022-11-23T01:19:07.7505832Z 
2022-11-23T01:19:07.7506054Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7506128Z Ran 1 test in 2.373s
2022-11-23T01:19:07.7506133Z 
2022-11-23T01:19:07.7506191Z OK
2022-11-23T01:19:07.7506197Z 
2022-11-23T01:19:07.7506267Z Generating XML reports...
2022-11-23T01:19:07.7506612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011359.xml
2022-11-23T01:19:07.7506844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7507116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7507237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7507516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7507649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7507657Z 
2022-11-23T01:19:07.7507726Z Running tests...
2022-11-23T01:19:07.7507910Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7508137Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7909
2022-11-23T01:19:07.7508292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7910
2022-11-23T01:19:07.7508447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7911
2022-11-23T01:19:07.7508718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7508839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7509117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7509248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7509427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7509686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7509806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7510088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7510224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7510400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7510671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7510791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7511071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7511192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7511365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7511691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7511989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7512282Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7512447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7512610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7512799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7512971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7513134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7513306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7522283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7522673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7522974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7523045Z ok (2.274s)
2022-11-23T01:19:07.7523052Z 
2022-11-23T01:19:07.7523249Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7523312Z Ran 1 test in 2.274s
2022-11-23T01:19:07.7523319Z 
2022-11-23T01:19:07.7523375Z OK
2022-11-23T01:19:07.7523380Z 
2022-11-23T01:19:07.7523462Z Generating XML reports...
2022-11-23T01:19:07.7523808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011403.xml
2022-11-23T01:19:07.7524043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7524320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7524442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7524724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7524850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7524865Z 
2022-11-23T01:19:07.7524924Z Running tests...
2022-11-23T01:19:07.7525118Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7525822Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:19:07.7525829Z 
2022-11-23T01:19:07.7526019Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7526090Z Ran 1 test in 0.253s
2022-11-23T01:19:07.7526095Z 
2022-11-23T01:19:07.7526163Z OK (skipped=1)
2022-11-23T01:19:07.7526168Z 
2022-11-23T01:19:07.7526249Z Generating XML reports...
2022-11-23T01:19:07.7526589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011407.xml
2022-11-23T01:19:07.7526820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7527172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7527298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7527579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7527710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7527715Z 
2022-11-23T01:19:07.7527784Z Running tests...
2022-11-23T01:19:07.7527976Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7528310Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s)
2022-11-23T01:19:07.7528349Z 
2022-11-23T01:19:07.7528540Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7528612Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7528617Z 
2022-11-23T01:19:07.7528673Z OK (skipped=1)
2022-11-23T01:19:07.7528678Z 
2022-11-23T01:19:07.7528765Z Generating XML reports...
2022-11-23T01:19:07.7529102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011409.xml
2022-11-23T01:19:07.7529332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7529607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7529822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7530108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7530245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7530251Z 
2022-11-23T01:19:07.7530320Z Running tests...
2022-11-23T01:19:07.7530502Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7530747Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7988
2022-11-23T01:19:07.7530903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7989
2022-11-23T01:19:07.7531055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7990
2022-11-23T01:19:07.7531328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7531448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7531734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7531868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7532034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7532312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7532432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7532709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7532843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7533020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7533291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7533413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7533693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7533848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7534022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7534319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7534618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7535069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7535290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7535451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7535615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7535668Z ok (2.171s)
2022-11-23T01:19:07.7535687Z 
2022-11-23T01:19:07.7535871Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7535944Z Ran 1 test in 2.171s
2022-11-23T01:19:07.7535949Z 
2022-11-23T01:19:07.7536008Z OK
2022-11-23T01:19:07.7536013Z 
2022-11-23T01:19:07.7536095Z Generating XML reports...
2022-11-23T01:19:07.7536436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011411.xml
2022-11-23T01:19:07.7536671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7536947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7537070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7537339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7537473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7537479Z 
2022-11-23T01:19:07.7537551Z Running tests...
2022-11-23T01:19:07.7537746Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7537961Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8041
2022-11-23T01:19:07.7538115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8042
2022-11-23T01:19:07.7538269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8043
2022-11-23T01:19:07.7538543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7538664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7538932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7539068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7539242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7539512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7539632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7539913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7540047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7540223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7540480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7540642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7540924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7541056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7541232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7541531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7541826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7542162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7542325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7542478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7542724Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7542966Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7543125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7543360Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7543605Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7543845Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7544276Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7544283Z 
2022-11-23T01:19:07.7544526Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7544777Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7545003Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7545236Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7545481Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7545729Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7545964Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7546201Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7546441Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7546690Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7546930Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7547168Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7547404Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7547642Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7547890Z STAGE:2022-11-23 01:14:16 8041:8041 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7548149Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7548390Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7548637Z STAGE:2022-11-23 01:14:16 8042:8042 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7548875Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7549113Z STAGE:2022-11-23 01:14:16 8043:8043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7549205Z ok (2.374s)
2022-11-23T01:19:07.7549211Z 
2022-11-23T01:19:07.7549405Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7549479Z Ran 1 test in 2.374s
2022-11-23T01:19:07.7549484Z 
2022-11-23T01:19:07.7549540Z OK
2022-11-23T01:19:07.7549545Z 
2022-11-23T01:19:07.7549626Z Generating XML reports...
2022-11-23T01:19:07.7549973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011414.xml
2022-11-23T01:19:07.7550205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7550481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7550592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7550871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7551005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7551010Z 
2022-11-23T01:19:07.7551079Z Running tests...
2022-11-23T01:19:07.7551274Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7551502Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8097
2022-11-23T01:19:07.7551656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8098
2022-11-23T01:19:07.7551806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8099
2022-11-23T01:19:07.7552066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7552185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7552462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7552594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7552768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7553037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7553160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7553433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7553566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7553730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7554029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7554299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7554421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7554696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7554887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7555062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7555360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7555657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7555808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7555999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7556159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7556224Z ok (2.074s)
2022-11-23T01:19:07.7556229Z 
2022-11-23T01:19:07.7556426Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7556505Z Ran 1 test in 2.075s
2022-11-23T01:19:07.7556511Z 
2022-11-23T01:19:07.7556568Z OK
2022-11-23T01:19:07.7556573Z 
2022-11-23T01:19:07.7556657Z Generating XML reports...
2022-11-23T01:19:07.7556989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011419.xml
2022-11-23T01:19:07.7557222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7557495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7557619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7557903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7558033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7558038Z 
2022-11-23T01:19:07.7558110Z Running tests...
2022-11-23T01:19:07.7558304Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7558486Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-11-23T01:19:07.7558492Z 
2022-11-23T01:19:07.7558671Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7558743Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7558748Z 
2022-11-23T01:19:07.7558814Z OK (skipped=1)
2022-11-23T01:19:07.7558819Z 
2022-11-23T01:19:07.7558901Z Generating XML reports...
2022-11-23T01:19:07.7559239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011422.xml
2022-11-23T01:19:07.7559468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7559738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7559861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7560140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7560261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7560267Z 
2022-11-23T01:19:07.7560335Z Running tests...
2022-11-23T01:19:07.7560524Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7560751Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8160
2022-11-23T01:19:07.7560906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8161
2022-11-23T01:19:07.7561056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8162
2022-11-23T01:19:07.7561351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7561477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7561744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7561876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7562049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7562318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7562466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7562743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7562876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7563053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7563322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7563432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7563709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7563844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7564020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7564316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7564614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7564906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7565070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7565231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7565380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7565553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7565730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7565901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7566198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7566490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7566777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7567021Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7567262Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7567493Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7567734Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7567970Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7568250Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7568505Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7568746Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7568998Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7569392Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7569431Z 
2022-11-23T01:19:07.7569672Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7569993Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7570233Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7570467Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7570710Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7570961Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7571201Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7571453Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7571689Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7571930Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7572168Z STAGE:2022-11-23 01:14:26 8160:8160 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7572404Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7572642Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7572890Z STAGE:2022-11-23 01:14:26 8161:8161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7573133Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7573387Z STAGE:2022-11-23 01:14:26 8162:8162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7573455Z ok (2.273s)
2022-11-23T01:19:07.7573461Z 
2022-11-23T01:19:07.7573658Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7573731Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7573736Z 
2022-11-23T01:19:07.7573786Z OK
2022-11-23T01:19:07.7573791Z 
2022-11-23T01:19:07.7573872Z Generating XML reports...
2022-11-23T01:19:07.7574213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011424.xml
2022-11-23T01:19:07.7574442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7574715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7574835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7575216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7575351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7575357Z 
2022-11-23T01:19:07.7575430Z Running tests...
2022-11-23T01:19:07.7575665Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7575892Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8225
2022-11-23T01:19:07.7576046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8226
2022-11-23T01:19:07.7576200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8227
2022-11-23T01:19:07.7576469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7576591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7576910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7577044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7577206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7577482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7577604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7577883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7578016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7578192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7578462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7578583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7578863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7578986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7579161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7579458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7579756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7580049Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7580216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7580380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7580539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7580710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7580872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7581041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7581334Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7581622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7581869Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7582164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7582432Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7582679Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7582931Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7583160Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7583410Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7583678Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7583913Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7584157Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7584407Z STAGE:2022-11-23 01:14:30 8226:8226 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7584645Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7584894Z STAGE:2022-11-23 01:14:30 8227:8227 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7584948Z ok (2.377s)
2022-11-23T01:19:07.7584965Z 
2022-11-23T01:19:07.7585149Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7585222Z Ran 1 test in 2.377s
2022-11-23T01:19:07.7585230Z 
2022-11-23T01:19:07.7585289Z OK
2022-11-23T01:19:07.7585294Z 
2022-11-23T01:19:07.7585376Z Generating XML reports...
2022-11-23T01:19:07.7585718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011428.xml
2022-11-23T01:19:07.7585949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7586224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7586345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7586615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7586748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7586754Z 
2022-11-23T01:19:07.7586822Z Running tests...
2022-11-23T01:19:07.7587016Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7587244Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8286
2022-11-23T01:19:07.7587398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8287
2022-11-23T01:19:07.7587549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8288
2022-11-23T01:19:07.7587821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7587942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7588210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7588344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7588519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7588790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7588910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7589191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7589350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7589524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7589783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7589904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7590182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7590355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7590529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7590826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7591128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7591421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7591585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7591736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7591895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7591962Z ok (2.174s)
2022-11-23T01:19:07.7591968Z 
2022-11-23T01:19:07.7592162Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7592235Z Ran 1 test in 2.174s
2022-11-23T01:19:07.7592240Z 
2022-11-23T01:19:07.7592299Z OK
2022-11-23T01:19:07.7592304Z 
2022-11-23T01:19:07.7592385Z Generating XML reports...
2022-11-23T01:19:07.7592725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011432.xml
2022-11-23T01:19:07.7592945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7593216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7593339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7593619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7593754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7593760Z 
2022-11-23T01:19:07.7593829Z Running tests...
2022-11-23T01:19:07.7594030Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7594720Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2022-11-23T01:19:07.7594726Z 
2022-11-23T01:19:07.7594920Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7594992Z Ran 1 test in 0.256s
2022-11-23T01:19:07.7594998Z 
2022-11-23T01:19:07.7595056Z OK (skipped=1)
2022-11-23T01:19:07.7595072Z 
2022-11-23T01:19:07.7595144Z Generating XML reports...
2022-11-23T01:19:07.7595485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011436.xml
2022-11-23T01:19:07.7595718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7596018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7596141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7596420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7596552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7596558Z 
2022-11-23T01:19:07.7596629Z Running tests...
2022-11-23T01:19:07.7596808Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7597030Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8349
2022-11-23T01:19:07.7597210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8350
2022-11-23T01:19:07.7597363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8351
2022-11-23T01:19:07.7597633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7597753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7598033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7598166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7598344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7598604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7598724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7599000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7599132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7599308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7599579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7599700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7599977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7600099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7600273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7600568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7600862Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7601149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7601312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7601471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7601629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7601802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7601966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7602139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7602457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7602750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7603040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7603106Z ok (2.175s)
2022-11-23T01:19:07.7603112Z 
2022-11-23T01:19:07.7603303Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7603375Z Ran 1 test in 2.175s
2022-11-23T01:19:07.7603381Z 
2022-11-23T01:19:07.7603465Z OK
2022-11-23T01:19:07.7603470Z 
2022-11-23T01:19:07.7603540Z Generating XML reports...
2022-11-23T01:19:07.7603878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011438.xml
2022-11-23T01:19:07.7604110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7604383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7604505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7604784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7604917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7604922Z 
2022-11-23T01:19:07.7604991Z Running tests...
2022-11-23T01:19:07.7605183Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7605394Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8408
2022-11-23T01:19:07.7605548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8409
2022-11-23T01:19:07.7605700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8410
2022-11-23T01:19:07.7605974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7606094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7606375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7606508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7606683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7606946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7607064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7607339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7607477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7607650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7607919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7608040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7608318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7608451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7608615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7608908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7609229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7609522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7609757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7609925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7610085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7610186Z ok (2.175s)
2022-11-23T01:19:07.7610192Z 
2022-11-23T01:19:07.7610390Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7610452Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7610457Z 
2022-11-23T01:19:07.7610516Z OK
2022-11-23T01:19:07.7610521Z 
2022-11-23T01:19:07.7610604Z Generating XML reports...
2022-11-23T01:19:07.7610946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011442.xml
2022-11-23T01:19:07.7611182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7611455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7611577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7611857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7611981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7611996Z 
2022-11-23T01:19:07.7612055Z Running tests...
2022-11-23T01:19:07.7612247Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7612470Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8461
2022-11-23T01:19:07.7612623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8462
2022-11-23T01:19:07.7612771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8463
2022-11-23T01:19:07.7613042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7613164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7613441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7613565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7613737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7614008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7614130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7614408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7614538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7614713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7615097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7615210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7615486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7615617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7615838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7616137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7616434Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7616729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7616896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7617095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7617245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7617311Z ok (2.275s)
2022-11-23T01:19:07.7617317Z 
2022-11-23T01:19:07.7617512Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7617586Z Ran 1 test in 2.275s
2022-11-23T01:19:07.7617592Z 
2022-11-23T01:19:07.7617649Z OK
2022-11-23T01:19:07.7617654Z 
2022-11-23T01:19:07.7617736Z Generating XML reports...
2022-11-23T01:19:07.7618079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011446.xml
2022-11-23T01:19:07.7618311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7618581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7618694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7618970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7619101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7619106Z 
2022-11-23T01:19:07.7619178Z Running tests...
2022-11-23T01:19:07.7619369Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7619604Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8514
2022-11-23T01:19:07.7619758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8515
2022-11-23T01:19:07.7619909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8516
2022-11-23T01:19:07.7620168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7620293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7620575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7620711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7620887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7621155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7621275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7621559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7621691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7621856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7622123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7622243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7622547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7622679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7622852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7623148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7623444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7623760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7623911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7624073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7624235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7624407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7624576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7624873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7625040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7625332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7625618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7625673Z ok (2.276s)
2022-11-23T01:19:07.7625678Z 
2022-11-23T01:19:07.7625875Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7625947Z Ran 1 test in 2.276s
2022-11-23T01:19:07.7625953Z 
2022-11-23T01:19:07.7626009Z OK
2022-11-23T01:19:07.7626014Z 
2022-11-23T01:19:07.7626098Z Generating XML reports...
2022-11-23T01:19:07.7626437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011450.xml
2022-11-23T01:19:07.7626669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7626940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7627049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7627329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7627466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7627472Z 
2022-11-23T01:19:07.7627542Z Running tests...
2022-11-23T01:19:07.7627739Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7627967Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8576
2022-11-23T01:19:07.7628117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8577
2022-11-23T01:19:07.7628266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8578
2022-11-23T01:19:07.7628536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7628647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7628924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7629085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7629261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7629534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7629654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7629934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7630095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7630268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7630525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7630643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7630918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7631048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7631220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7631515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7631809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7632107Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7632270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7632423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7632583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7632754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7632926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7633096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7633394Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7633687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7633980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7634035Z ok (2.273s)
2022-11-23T01:19:07.7634051Z 
2022-11-23T01:19:07.7634234Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7634306Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7634312Z 
2022-11-23T01:19:07.7634371Z OK
2022-11-23T01:19:07.7634377Z 
2022-11-23T01:19:07.7634457Z Generating XML reports...
2022-11-23T01:19:07.7634798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011454.xml
2022-11-23T01:19:07.7635031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7635304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7635425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7635729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7635862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7635868Z 
2022-11-23T01:19:07.7635937Z Running tests...
2022-11-23T01:19:07.7636129Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7636448Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2022-11-23T01:19:07.7636454Z 
2022-11-23T01:19:07.7636641Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7636755Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7636761Z 
2022-11-23T01:19:07.7636829Z OK (skipped=1)
2022-11-23T01:19:07.7636834Z 
2022-11-23T01:19:07.7636916Z Generating XML reports...
2022-11-23T01:19:07.7637242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011458.xml
2022-11-23T01:19:07.7637474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7637744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7637869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7638145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7638276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7638281Z 
2022-11-23T01:19:07.7638352Z Running tests...
2022-11-23T01:19:07.7638539Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7638756Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8645
2022-11-23T01:19:07.7638899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8646
2022-11-23T01:19:07.7639052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8647
2022-11-23T01:19:07.7639327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7639448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7639725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7639857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7640033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7640302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7640410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7640689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7640821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7640994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7641259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7641380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7641660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7641794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7641967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7642278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7642572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7642860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7643024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7643187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7643375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7643439Z ok (2.275s)
2022-11-23T01:19:07.7643445Z 
2022-11-23T01:19:07.7643639Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7643699Z Ran 1 test in 2.275s
2022-11-23T01:19:07.7643716Z 
2022-11-23T01:19:07.7643763Z OK
2022-11-23T01:19:07.7643767Z 
2022-11-23T01:19:07.7643849Z Generating XML reports...
2022-11-23T01:19:07.7644189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011459.xml
2022-11-23T01:19:07.7644420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7644693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7644815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7645093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7645229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7645235Z 
2022-11-23T01:19:07.7645294Z Running tests...
2022-11-23T01:19:07.7645487Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7645705Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8698
2022-11-23T01:19:07.7645858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8699
2022-11-23T01:19:07.7646009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8700
2022-11-23T01:19:07.7646277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7646398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7646679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7646813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7646976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7647248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7647368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7647644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7647777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7647952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7648219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7648342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7648609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7648773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7648947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7649242Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7649534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7649898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7650099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7650261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7650420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7650474Z ok (2.277s)
2022-11-23T01:19:07.7650482Z 
2022-11-23T01:19:07.7650679Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7650752Z Ran 1 test in 2.278s
2022-11-23T01:19:07.7650757Z 
2022-11-23T01:19:07.7650817Z OK
2022-11-23T01:19:07.7650822Z 
2022-11-23T01:19:07.7650906Z Generating XML reports...
2022-11-23T01:19:07.7651247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011503.xml
2022-11-23T01:19:07.7651478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7651755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7651876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7652142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7652275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7652281Z 
2022-11-23T01:19:07.7652351Z Running tests...
2022-11-23T01:19:07.7652544Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7652784Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8751
2022-11-23T01:19:07.7652936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8752
2022-11-23T01:19:07.7653087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8753
2022-11-23T01:19:07.7653359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7653469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7653745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7653879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7654053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7654323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7654446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7654726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7654967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7655143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7655403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7655568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7655848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7655981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7656154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7656452Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7656748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7657113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7657276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7657429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7657675Z STAGE:2022-11-23 01:15:09 8753:8753 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7657913Z STAGE:2022-11-23 01:15:09 8752:8752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7658074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7658315Z STAGE:2022-11-23 01:15:09 8751:8751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7658564Z STAGE:2022-11-23 01:15:09 8751:8751 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7658821Z STAGE:2022-11-23 01:15:09 8751:8751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7659066Z STAGE:2022-11-23 01:15:09 8752:8752 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7659308Z STAGE:2022-11-23 01:15:09 8752:8752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7659550Z STAGE:2022-11-23 01:15:09 8753:8753 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7659799Z STAGE:2022-11-23 01:15:09 8753:8753 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7659864Z ok (2.276s)
2022-11-23T01:19:07.7659870Z 
2022-11-23T01:19:07.7660066Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7660137Z Ran 1 test in 2.277s
2022-11-23T01:19:07.7660145Z 
2022-11-23T01:19:07.7660203Z OK
2022-11-23T01:19:07.7660208Z 
2022-11-23T01:19:07.7660288Z Generating XML reports...
2022-11-23T01:19:07.7660626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011507.xml
2022-11-23T01:19:07.7660848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7661123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7661243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7661521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7661651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7661656Z 
2022-11-23T01:19:07.7661727Z Running tests...
2022-11-23T01:19:07.7661921Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7662156Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8807
2022-11-23T01:19:07.7662309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8808
2022-11-23T01:19:07.7662475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8809
2022-11-23T01:19:07.7662747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7662867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7663147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7663277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7663448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7663744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7663863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7664132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7664266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7664440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7664708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7664825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7665099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7665231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7665405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7665702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7665991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7666284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7666447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7666690Z STAGE:2022-11-23 01:15:13 8808:8808 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7666849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7667010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7667250Z STAGE:2022-11-23 01:15:13 8807:8807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7667486Z STAGE:2022-11-23 01:15:13 8809:8809 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7667727Z STAGE:2022-11-23 01:15:13 8809:8809 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7667968Z STAGE:2022-11-23 01:15:13 8809:8809 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7668210Z STAGE:2022-11-23 01:15:13 8808:8808 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7668459Z STAGE:2022-11-23 01:15:13 8808:8808 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7668702Z STAGE:2022-11-23 01:15:13 8807:8807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7668955Z STAGE:2022-11-23 01:15:13 8807:8807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7669021Z ok (2.273s)
2022-11-23T01:19:07.7669026Z 
2022-11-23T01:19:07.7669217Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7669289Z Ran 1 test in 2.274s
2022-11-23T01:19:07.7669295Z 
2022-11-23T01:19:07.7669368Z OK
2022-11-23T01:19:07.7669383Z 
2022-11-23T01:19:07.7669454Z Generating XML reports...
2022-11-23T01:19:07.7669792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011511.xml
2022-11-23T01:19:07.7670023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7670295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7670414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7670721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7670853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7670858Z 
2022-11-23T01:19:07.7670926Z Running tests...
2022-11-23T01:19:07.7671110Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7671447Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:19:07.7671453Z 
2022-11-23T01:19:07.7671642Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7671714Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7671720Z 
2022-11-23T01:19:07.7671788Z OK (skipped=1)
2022-11-23T01:19:07.7671793Z 
2022-11-23T01:19:07.7671872Z Generating XML reports...
2022-11-23T01:19:07.7672208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011515.xml
2022-11-23T01:19:07.7672441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7672713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7672824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7673104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7673235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7673240Z 
2022-11-23T01:19:07.7673310Z Running tests...
2022-11-23T01:19:07.7673536Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7673972Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:19:07.7673981Z 
2022-11-23T01:19:07.7674171Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7674245Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7674250Z 
2022-11-23T01:19:07.7674316Z OK (skipped=1)
2022-11-23T01:19:07.7674321Z 
2022-11-23T01:19:07.7674391Z Generating XML reports...
2022-11-23T01:19:07.7674730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011517.xml
2022-11-23T01:19:07.7674959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7675231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7675350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7675626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7675760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7675766Z 
2022-11-23T01:19:07.7675833Z Running tests...
2022-11-23T01:19:07.7676015Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7676295Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8883
2022-11-23T01:19:07.7676451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8884
2022-11-23T01:19:07.7676599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8885
2022-11-23T01:19:07.7676873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7676994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7677271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7677432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7677607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7677868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7677987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7678262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7678394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7678570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7678868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7679140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7679261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7679541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7679664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7679836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7680129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7680422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7680583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7680747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7680910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7681170Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:19:07.7681331Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:19:07.7681387Z ok (4.186s)
2022-11-23T01:19:07.7681392Z 
2022-11-23T01:19:07.7681589Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7681659Z Ran 1 test in 4.186s
2022-11-23T01:19:07.7681665Z 
2022-11-23T01:19:07.7681721Z OK
2022-11-23T01:19:07.7681727Z 
2022-11-23T01:19:07.7681807Z Generating XML reports...
2022-11-23T01:19:07.7682148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011519.xml
2022-11-23T01:19:07.7682379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7682652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7682790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7683073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7683204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7683209Z 
2022-11-23T01:19:07.7683279Z Running tests...
2022-11-23T01:19:07.7683474Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7683708Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8936
2022-11-23T01:19:07.7683903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8937
2022-11-23T01:19:07.7684052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8938
2022-11-23T01:19:07.7684416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7684528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7684810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7684943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7685119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7685389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7685511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7685792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7685923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7686099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7686358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7686482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7686763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7686896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7687068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7687363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7687655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7687948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7688110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7688265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7688422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7688586Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:19:07.7688842Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:19:07.7688908Z ok (4.378s)
2022-11-23T01:19:07.7688914Z 
2022-11-23T01:19:07.7689111Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7689217Z Ran 1 test in 4.379s
2022-11-23T01:19:07.7689223Z 
2022-11-23T01:19:07.7689281Z OK
2022-11-23T01:19:07.7689285Z 
2022-11-23T01:19:07.7689356Z Generating XML reports...
2022-11-23T01:19:07.7689758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011525.xml
2022-11-23T01:19:07.7689989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7690260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7690412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7690694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7690825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7690830Z 
2022-11-23T01:19:07.7690901Z Running tests...
2022-11-23T01:19:07.7691097Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7691330Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8989
2022-11-23T01:19:07.7691483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8990
2022-11-23T01:19:07.7691634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8991
2022-11-23T01:19:07.7691904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7692024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7692301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7692435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7692612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7692881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7692991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7693266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7693399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7693572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7693845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7693967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7694246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7694377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7694541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7694972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7695395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7695696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7695859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7696023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7696242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7696419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7696593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7696753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7697049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7697378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7697665Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7697821Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T01:19:07.7697941Z No ranks successfully processed in monitoredBarrier.
2022-11-23T01:19:07.7698098Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T01:19:07.7698168Z ok (2.376s)
2022-11-23T01:19:07.7698174Z 
2022-11-23T01:19:07.7698368Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7698431Z Ran 1 test in 2.376s
2022-11-23T01:19:07.7698436Z 
2022-11-23T01:19:07.7698493Z OK
2022-11-23T01:19:07.7698498Z 
2022-11-23T01:19:07.7698581Z Generating XML reports...
2022-11-23T01:19:07.7698924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011531.xml
2022-11-23T01:19:07.7699157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7699427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7699551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7699831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7699953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7699970Z 
2022-11-23T01:19:07.7700028Z Running tests...
2022-11-23T01:19:07.7700220Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7700468Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9051
2022-11-23T01:19:07.7700623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9052
2022-11-23T01:19:07.7700773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9053
2022-11-23T01:19:07.7701047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7701165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7701445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7701566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7701739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7702011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7702132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7702408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7702541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7702752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7703024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7703145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7703413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7703543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7703721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7704046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7704338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7704628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7704791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7704952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7705111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7705272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7705445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7705741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7705910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7706203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7706492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7706654Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:19:07.7707026Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:444: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-11-23T01:19:07.7707156Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-11-23T01:19:07.7707210Z ok (2.374s)
2022-11-23T01:19:07.7707216Z 
2022-11-23T01:19:07.7707412Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7707484Z Ran 1 test in 2.375s
2022-11-23T01:19:07.7707490Z 
2022-11-23T01:19:07.7707547Z OK
2022-11-23T01:19:07.7707554Z 
2022-11-23T01:19:07.7707635Z Generating XML reports...
2022-11-23T01:19:07.7707974Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011535.xml
2022-11-23T01:19:07.7708203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7708476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7708586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7708871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7709003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7709009Z 
2022-11-23T01:19:07.7709081Z Running tests...
2022-11-23T01:19:07.7709271Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7709542Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9110
2022-11-23T01:19:07.7709695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9111
2022-11-23T01:19:07.7709849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9112
2022-11-23T01:19:07.7710121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7710231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7710536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7710669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7710842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7711111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7711232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7711508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7711640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7711813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7712101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7712370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7712488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7712764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7712893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7713066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7713364Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7713658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7713825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7713975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7714134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7714296Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:19:07.7714453Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:19:07.7714611Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:19:07.7714674Z ok (2.379s)
2022-11-23T01:19:07.7714679Z 
2022-11-23T01:19:07.7714873Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7714944Z Ran 1 test in 2.379s
2022-11-23T01:19:07.7714951Z 
2022-11-23T01:19:07.7714999Z OK
2022-11-23T01:19:07.7715004Z 
2022-11-23T01:19:07.7715085Z Generating XML reports...
2022-11-23T01:19:07.7715423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011539.xml
2022-11-23T01:19:07.7715653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7715951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7716072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7716351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7716484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7716490Z 
2022-11-23T01:19:07.7716559Z Running tests...
2022-11-23T01:19:07.7716741Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7717074Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:19:07.7717080Z 
2022-11-23T01:19:07.7717268Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7717341Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7717346Z 
2022-11-23T01:19:07.7717417Z OK (skipped=1)
2022-11-23T01:19:07.7717422Z 
2022-11-23T01:19:07.7717502Z Generating XML reports...
2022-11-23T01:19:07.7717839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011543.xml
2022-11-23T01:19:07.7718069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7718329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7718452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7718736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7718868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7718873Z 
2022-11-23T01:19:07.7718941Z Running tests...
2022-11-23T01:19:07.7719133Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7719438Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:19:07.7719444Z 
2022-11-23T01:19:07.7719633Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7719703Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7719708Z 
2022-11-23T01:19:07.7719765Z OK (skipped=1)
2022-11-23T01:19:07.7719780Z 
2022-11-23T01:19:07.7719850Z Generating XML reports...
2022-11-23T01:19:07.7720191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011545.xml
2022-11-23T01:19:07.7720423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7720692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7720813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7721087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7721219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7721225Z 
2022-11-23T01:19:07.7721292Z Running tests...
2022-11-23T01:19:07.7721474Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7721774Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:19:07.7721782Z 
2022-11-23T01:19:07.7721971Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7722045Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7722051Z 
2022-11-23T01:19:07.7722119Z OK (skipped=1)
2022-11-23T01:19:07.7722124Z 
2022-11-23T01:19:07.7722201Z Generating XML reports...
2022-11-23T01:19:07.7722566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011546.xml
2022-11-23T01:19:07.7722795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7723068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7723177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7723457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7723617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7723622Z 
2022-11-23T01:19:07.7723691Z Running tests...
2022-11-23T01:19:07.7723884Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7724178Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:19:07.7724186Z 
2022-11-23T01:19:07.7724376Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7724447Z Ran 1 test in 0.003s
2022-11-23T01:19:07.7724452Z 
2022-11-23T01:19:07.7724518Z OK (skipped=1)
2022-11-23T01:19:07.7724523Z 
2022-11-23T01:19:07.7724592Z Generating XML reports...
2022-11-23T01:19:07.7724926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011548.xml
2022-11-23T01:19:07.7725154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7725428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7725548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7725827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7725963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7725968Z 
2022-11-23T01:19:07.7726036Z Running tests...
2022-11-23T01:19:07.7726215Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7726431Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T01:19:07.7726436Z 
2022-11-23T01:19:07.7726628Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7726702Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7726709Z 
2022-11-23T01:19:07.7726777Z OK (skipped=1)
2022-11-23T01:19:07.7726782Z 
2022-11-23T01:19:07.7726863Z Generating XML reports...
2022-11-23T01:19:07.7727196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011550.xml
2022-11-23T01:19:07.7727424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7727696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7727806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7728084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7728216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7728222Z 
2022-11-23T01:19:07.7728290Z Running tests...
2022-11-23T01:19:07.7728479Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7728659Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:19:07.7728665Z 
2022-11-23T01:19:07.7728851Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7728921Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7728926Z 
2022-11-23T01:19:07.7729022Z OK (skipped=1)
2022-11-23T01:19:07.7729027Z 
2022-11-23T01:19:07.7729098Z Generating XML reports...
2022-11-23T01:19:07.7729432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011551.xml
2022-11-23T01:19:07.7729663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7730012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7730134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7730460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7730592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7730598Z 
2022-11-23T01:19:07.7730666Z Running tests...
2022-11-23T01:19:07.7730850Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7731045Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:19:07.7731050Z 
2022-11-23T01:19:07.7731241Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7731313Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7731318Z 
2022-11-23T01:19:07.7731389Z OK (skipped=1)
2022-11-23T01:19:07.7731394Z 
2022-11-23T01:19:07.7731474Z Generating XML reports...
2022-11-23T01:19:07.7731809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011553.xml
2022-11-23T01:19:07.7732039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7732307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7732418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7732696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7732829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7732834Z 
2022-11-23T01:19:07.7732904Z Running tests...
2022-11-23T01:19:07.7733092Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7733315Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:19:07.7733323Z 
2022-11-23T01:19:07.7733511Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7733585Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7733591Z 
2022-11-23T01:19:07.7733658Z OK (skipped=1)
2022-11-23T01:19:07.7733663Z 
2022-11-23T01:19:07.7733733Z Generating XML reports...
2022-11-23T01:19:07.7734069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011554.xml
2022-11-23T01:19:07.7734297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7734567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7734687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7735062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7735199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7735205Z 
2022-11-23T01:19:07.7735274Z Running tests...
2022-11-23T01:19:07.7735468Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7735762Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9243
2022-11-23T01:19:07.7735920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9244
2022-11-23T01:19:07.7736074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9245
2022-11-23T01:19:07.7736349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7736473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7736753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7736926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7737103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7737377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7737486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7737761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7737894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7738070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7738343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7738468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7738750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7738881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7739049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7739345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7739639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7739927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7740091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7740256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7740418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7740511Z skip: CUDA is not available. (2.172s)
2022-11-23T01:19:07.7740516Z 
2022-11-23T01:19:07.7740710Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7740770Z Ran 1 test in 2.172s
2022-11-23T01:19:07.7740776Z 
2022-11-23T01:19:07.7740843Z OK (skipped=1)
2022-11-23T01:19:07.7740848Z 
2022-11-23T01:19:07.7740934Z Generating XML reports...
2022-11-23T01:19:07.7741269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011556.xml
2022-11-23T01:19:07.7741501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7741769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7741891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7742172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7742332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7742338Z 
2022-11-23T01:19:07.7742397Z Running tests...
2022-11-23T01:19:07.7742587Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7742839Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9296
2022-11-23T01:19:07.7742992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9297
2022-11-23T01:19:07.7743143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9298
2022-11-23T01:19:07.7743440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7743560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7743833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7743955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7744130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7744398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7744518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7744796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7744930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7745106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7745374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7745495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7745761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7745889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7746062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7746358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7746657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7746948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7747113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7747278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7747438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7747520Z skip: CUDA is not available. (2.071s)
2022-11-23T01:19:07.7747525Z 
2022-11-23T01:19:07.7747717Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7747788Z Ran 1 test in 2.071s
2022-11-23T01:19:07.7747793Z 
2022-11-23T01:19:07.7747862Z OK (skipped=1)
2022-11-23T01:19:07.7747867Z 
2022-11-23T01:19:07.7747946Z Generating XML reports...
2022-11-23T01:19:07.7748287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011600.xml
2022-11-23T01:19:07.7748516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7748787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7748928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7749210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7749340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7749346Z 
2022-11-23T01:19:07.7749415Z Running tests...
2022-11-23T01:19:07.7749606Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7749807Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:19:07.7749842Z 
2022-11-23T01:19:07.7750031Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7750102Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7750108Z 
2022-11-23T01:19:07.7750175Z OK (skipped=1)
2022-11-23T01:19:07.7750180Z 
2022-11-23T01:19:07.7750250Z Generating XML reports...
2022-11-23T01:19:07.7750587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011604.xml
2022-11-23T01:19:07.7750829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7751100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7751221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7751499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7751632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7751638Z 
2022-11-23T01:19:07.7751706Z Running tests...
2022-11-23T01:19:07.7751897Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7752103Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s)
2022-11-23T01:19:07.7752119Z 
2022-11-23T01:19:07.7752298Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7752370Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7752376Z 
2022-11-23T01:19:07.7752443Z OK (skipped=1)
2022-11-23T01:19:07.7752448Z 
2022-11-23T01:19:07.7752529Z Generating XML reports...
2022-11-23T01:19:07.7752865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011605.xml
2022-11-23T01:19:07.7753095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7753365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7753485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7753753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7753889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7753895Z 
2022-11-23T01:19:07.7753963Z Running tests...
2022-11-23T01:19:07.7754153Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7754853Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7754861Z 
2022-11-23T01:19:07.7755052Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7755124Z Ran 1 test in 0.254s
2022-11-23T01:19:07.7755130Z 
2022-11-23T01:19:07.7755199Z OK (skipped=1)
2022-11-23T01:19:07.7755204Z 
2022-11-23T01:19:07.7755314Z Generating XML reports...
2022-11-23T01:19:07.7755656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011607.xml
2022-11-23T01:19:07.7755874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7756144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7756266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7756574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7756707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7756713Z 
2022-11-23T01:19:07.7756780Z Running tests...
2022-11-23T01:19:07.7756969Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7757213Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9379
2022-11-23T01:19:07.7757355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9380
2022-11-23T01:19:07.7757506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9381
2022-11-23T01:19:07.7757776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7757896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7758177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7758311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7758486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7758759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7758877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7759144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7759276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7759453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7759724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7759845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7760128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7760260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7760434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7760731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7761018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7761307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7761471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7761635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7761796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7761957Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:19:07.7761963Z 
2022-11-23T01:19:07.7762159Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7762232Z Ran 1 test in 2.075s
2022-11-23T01:19:07.7762237Z 
2022-11-23T01:19:07.7762293Z OK (skipped=1)
2022-11-23T01:19:07.7762311Z 
2022-11-23T01:19:07.7762381Z Generating XML reports...
2022-11-23T01:19:07.7762723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011609.xml
2022-11-23T01:19:07.7762953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7763253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7763374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7763656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7763792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7763797Z 
2022-11-23T01:19:07.7763866Z Running tests...
2022-11-23T01:19:07.7764050Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7764288Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9432
2022-11-23T01:19:07.7764441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9433
2022-11-23T01:19:07.7764597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9434
2022-11-23T01:19:07.7764870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7764992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7765275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7765409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7765584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7765846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7765966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7766243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7766377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7766552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7766847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7767113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7767234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7767501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7767630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7767803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7768102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7768396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7768590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7768753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7768914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7769013Z skip: Need at least 2 CUDA devices (2.177s)
2022-11-23T01:19:07.7769019Z 
2022-11-23T01:19:07.7769202Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7769274Z Ran 1 test in 2.177s
2022-11-23T01:19:07.7769279Z 
2022-11-23T01:19:07.7769376Z OK (skipped=1)
2022-11-23T01:19:07.7769381Z 
2022-11-23T01:19:07.7769462Z Generating XML reports...
2022-11-23T01:19:07.7769871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011613.xml
2022-11-23T01:19:07.7770103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7770376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7770501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7770781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7770903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7770909Z 
2022-11-23T01:19:07.7770978Z Running tests...
2022-11-23T01:19:07.7771170Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7771423Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9485
2022-11-23T01:19:07.7771576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9486
2022-11-23T01:19:07.7771727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9487
2022-11-23T01:19:07.7771998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7772121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7772397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7772521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7772696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7772968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7773091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7773370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7773502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7773675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7773942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7774052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7774328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7774461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7774633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7775035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7775380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7775675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7775837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7775999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7776148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7776298Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:19:07.7776304Z 
2022-11-23T01:19:07.7776501Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7776575Z Ran 1 test in 2.176s
2022-11-23T01:19:07.7776581Z 
2022-11-23T01:19:07.7776648Z OK (skipped=1)
2022-11-23T01:19:07.7776654Z 
2022-11-23T01:19:07.7776737Z Generating XML reports...
2022-11-23T01:19:07.7777081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011617.xml
2022-11-23T01:19:07.7777310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7777583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7777694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7777972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7778107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7778112Z 
2022-11-23T01:19:07.7778179Z Running tests...
2022-11-23T01:19:07.7778376Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7779078Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7779085Z 
2022-11-23T01:19:07.7779275Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7779348Z Ran 1 test in 0.255s
2022-11-23T01:19:07.7779353Z 
2022-11-23T01:19:07.7779424Z OK (skipped=1)
2022-11-23T01:19:07.7779431Z 
2022-11-23T01:19:07.7779513Z Generating XML reports...
2022-11-23T01:19:07.7779842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011621.xml
2022-11-23T01:19:07.7780072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7780344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7780466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7780743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7780875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7780881Z 
2022-11-23T01:19:07.7780950Z Running tests...
2022-11-23T01:19:07.7781141Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7781860Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.251s)
2022-11-23T01:19:07.7781898Z 
2022-11-23T01:19:07.7782089Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7782151Z Ran 1 test in 0.251s
2022-11-23T01:19:07.7782157Z 
2022-11-23T01:19:07.7782223Z OK (skipped=1)
2022-11-23T01:19:07.7782228Z 
2022-11-23T01:19:07.7782309Z Generating XML reports...
2022-11-23T01:19:07.7782642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011622.xml
2022-11-23T01:19:07.7782872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7783173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7783293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7783568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7783688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7783705Z 
2022-11-23T01:19:07.7783762Z Running tests...
2022-11-23T01:19:07.7783951Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7784213Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9558
2022-11-23T01:19:07.7784365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9559
2022-11-23T01:19:07.7784515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9560
2022-11-23T01:19:07.7784787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7784908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7785188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7785310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7785482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7785754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7785873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7786150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7786287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7786460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7786731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7786852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7787117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7787249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7787422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7787720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7788019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7788310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7788500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7788663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7788824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7788915Z skip: Need at least 4 CUDA devices (2.072s)
2022-11-23T01:19:07.7788921Z 
2022-11-23T01:19:07.7789115Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7789188Z Ran 1 test in 2.072s
2022-11-23T01:19:07.7789193Z 
2022-11-23T01:19:07.7789260Z OK (skipped=1)
2022-11-23T01:19:07.7789295Z 
2022-11-23T01:19:07.7789377Z Generating XML reports...
2022-11-23T01:19:07.7789718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011624.xml
2022-11-23T01:19:07.7789946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7790221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7790333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7790613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7790743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7790749Z 
2022-11-23T01:19:07.7790818Z Running tests...
2022-11-23T01:19:07.7791009Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7791292Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9611
2022-11-23T01:19:07.7791443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9612
2022-11-23T01:19:07.7791597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9613
2022-11-23T01:19:07.7791865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7791974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7792249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7792379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7792552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7792826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7792947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7793225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7793359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7793523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7793791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7793909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7794185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7794317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7794488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7794785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7795111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7795400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7795563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7795714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7795873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7796001Z skip: Need at least 4 CUDA devices (2.075s)
2022-11-23T01:19:07.7796007Z 
2022-11-23T01:19:07.7796201Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7796273Z Ran 1 test in 2.076s
2022-11-23T01:19:07.7796278Z 
2022-11-23T01:19:07.7796346Z OK (skipped=1)
2022-11-23T01:19:07.7796352Z 
2022-11-23T01:19:07.7796435Z Generating XML reports...
2022-11-23T01:19:07.7796777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011628.xml
2022-11-23T01:19:07.7796995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7797265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7797387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7797665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7797798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7797804Z 
2022-11-23T01:19:07.7797871Z Running tests...
2022-11-23T01:19:07.7798062Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7798764Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:19:07.7798771Z 
2022-11-23T01:19:07.7798961Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7799031Z Ran 1 test in 0.254s
2022-11-23T01:19:07.7799036Z 
2022-11-23T01:19:07.7799094Z OK (skipped=1)
2022-11-23T01:19:07.7799101Z 
2022-11-23T01:19:07.7799181Z Generating XML reports...
2022-11-23T01:19:07.7799517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011632.xml
2022-11-23T01:19:07.7799747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7800022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7800142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7800418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7800552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7800557Z 
2022-11-23T01:19:07.7800625Z Running tests...
2022-11-23T01:19:07.7800804Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7801039Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9674
2022-11-23T01:19:07.7801189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9675
2022-11-23T01:19:07.7801339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9676
2022-11-23T01:19:07.7801640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7801761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7802040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7802171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7802335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7802634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7802752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7803027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7803158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7803332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7803602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7803721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7803997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7804120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7804295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7804592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7804890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7805179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7805340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7805502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7805663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7805837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7805997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7806167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7806460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7806748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7807035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7807437Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7807445Z 
2022-11-23T01:19:07.7807686Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7807930Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7808217Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7808471Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7808702Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7808954Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7809205Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7809443Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7809782Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7810021Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7810264Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7810513Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7810754Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7810978Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7811228Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7811478Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7811718Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7811952Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7812190Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7812430Z STAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7812846Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:16:36 9676:9676 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7812853Z 
2022-11-23T01:19:07.7813091Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7813331Z STAGE:2022-11-23 01:16:36 9675:9675 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7813584Z STAGE:2022-11-23 01:16:36 9674:9674 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7813648Z ok (2.271s)
2022-11-23T01:19:07.7813653Z 
2022-11-23T01:19:07.7813845Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7813920Z Ran 1 test in 2.272s
2022-11-23T01:19:07.7813926Z 
2022-11-23T01:19:07.7813982Z OK
2022-11-23T01:19:07.7813987Z 
2022-11-23T01:19:07.7814067Z Generating XML reports...
2022-11-23T01:19:07.7814406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011634.xml
2022-11-23T01:19:07.7814636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7815009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7815133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7815413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7815547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7815552Z 
2022-11-23T01:19:07.7815622Z Running tests...
2022-11-23T01:19:07.7815867Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7816099Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9739
2022-11-23T01:19:07.7816253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9740
2022-11-23T01:19:07.7816394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9741
2022-11-23T01:19:07.7816665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7816828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7817111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7817245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7817421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7817692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7817814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7818093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7818214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7818393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7818660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7818780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7819058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7819191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7819363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7819662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7819959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7820242Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7820404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7820564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7820726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7820898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7821070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7821364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7821530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7821823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7822104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7822373Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7822616Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7822854Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7823095Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7823511Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7823559Z 
2022-11-23T01:19:07.7823800Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7824049Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7824301Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7824528Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7824761Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7824994Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7825234Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7825483Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7825723Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7825957Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7826206Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7826457Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7826682Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7826915Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7827147Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7827554Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7827560Z 
2022-11-23T01:19:07.7827800Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7828049Z STAGE:2022-11-23 01:16:40 9739:9739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7828294Z STAGE:2022-11-23 01:16:40 9740:9740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7828542Z STAGE:2022-11-23 01:16:40 9741:9741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7828606Z ok (2.272s)
2022-11-23T01:19:07.7828612Z 
2022-11-23T01:19:07.7828806Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7828868Z Ran 1 test in 2.272s
2022-11-23T01:19:07.7828873Z 
2022-11-23T01:19:07.7828933Z OK
2022-11-23T01:19:07.7828938Z 
2022-11-23T01:19:07.7829020Z Generating XML reports...
2022-11-23T01:19:07.7829362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011638.xml
2022-11-23T01:19:07.7829590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7829888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7830013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7830292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7830415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7830430Z 
2022-11-23T01:19:07.7830488Z Running tests...
2022-11-23T01:19:07.7830681Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7830950Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9804
2022-11-23T01:19:07.7831104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9805
2022-11-23T01:19:07.7831253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9806
2022-11-23T01:19:07.7831527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7831648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7831925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7832048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7832223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7832495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7832615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7832895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7833028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7833199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7833465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7833576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7833851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7833983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7834159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7834455Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7834749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7835038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7835200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7835360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7835508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7835683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7835853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7836024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7836343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7836631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7836918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7837160Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7837398Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7837664Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7837898Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7838143Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7838378Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7838627Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7838879Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7839126Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7839362Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7839593Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7839817Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7840055Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7840305Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7840710Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7840716Z 
2022-11-23T01:19:07.7840966Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7841220Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7841453Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7841687Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7841922Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7842165Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7842390Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7842624Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7842873Z STAGE:2022-11-23 01:16:44 9804:9804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7843121Z STAGE:2022-11-23 01:16:44 9805:9805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7843370Z STAGE:2022-11-23 01:16:44 9806:9806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7843434Z ok (2.371s)
2022-11-23T01:19:07.7843439Z 
2022-11-23T01:19:07.7843659Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7843734Z Ran 1 test in 2.371s
2022-11-23T01:19:07.7843739Z 
2022-11-23T01:19:07.7843785Z OK
2022-11-23T01:19:07.7843799Z 
2022-11-23T01:19:07.7843869Z Generating XML reports...
2022-11-23T01:19:07.7844207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011642.xml
2022-11-23T01:19:07.7844436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7844710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7844857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7845134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7845268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7845274Z 
2022-11-23T01:19:07.7845345Z Running tests...
2022-11-23T01:19:07.7845528Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7845759Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9869
2022-11-23T01:19:07.7845909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9870
2022-11-23T01:19:07.7846058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9871
2022-11-23T01:19:07.7846326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7846449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7846725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7846855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7847032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7847314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7847585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7847706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7847983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7848115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7848290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7848564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7848686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7848955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7849087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7849259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7849551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7849921Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7850087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7850247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7850438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7850616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7850777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7850949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7851242Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7851559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7851843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7852089Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7852329Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7852568Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7852813Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7853054Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7853298Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7853534Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7853957Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7853963Z 
2022-11-23T01:19:07.7854200Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7854433Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7854668Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7855015Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7855271Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7855513Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7855742Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7855995Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7856245Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7856480Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7856714Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7856948Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7857189Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7857439Z STAGE:2022-11-23 01:16:48 9871:9871 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7857678Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7857951Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7858207Z STAGE:2022-11-23 01:16:48 9869:9869 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7858456Z STAGE:2022-11-23 01:16:48 9870:9870 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7858521Z ok (2.372s)
2022-11-23T01:19:07.7858527Z 
2022-11-23T01:19:07.7858718Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7858794Z Ran 1 test in 2.372s
2022-11-23T01:19:07.7858834Z 
2022-11-23T01:19:07.7858894Z OK
2022-11-23T01:19:07.7858899Z 
2022-11-23T01:19:07.7858982Z Generating XML reports...
2022-11-23T01:19:07.7859313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011646.xml
2022-11-23T01:19:07.7859547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7859823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7859946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7860223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7860355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7860361Z 
2022-11-23T01:19:07.7860431Z Running tests...
2022-11-23T01:19:07.7860626Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7860860Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9934
2022-11-23T01:19:07.7861002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9935
2022-11-23T01:19:07.7861155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9936
2022-11-23T01:19:07.7861427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7861549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7861826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7861957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7862130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7862400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7862521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7862787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7862922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7863099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7863369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7863490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7863766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7863898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7864071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7864357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7864686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7864984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7865148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7865307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7865481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7865709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7865866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7866033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7866318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7866610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7866902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7867146Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7867392Z STAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7867799Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7867806Z 
2022-11-23T01:19:07.7868233Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7868240Z 
2022-11-23T01:19:07.7868478Z STAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7868715Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7868959Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7869204Z STAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7869445Z STAGE:2022-11-23 01:16:52 9935:9935 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7869696Z STAGE:2022-11-23 01:16:52 9936:9936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7869762Z ok (2.375s)
2022-11-23T01:19:07.7869770Z 
2022-11-23T01:19:07.7869965Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7870037Z Ran 1 test in 2.375s
2022-11-23T01:19:07.7870042Z 
2022-11-23T01:19:07.7870098Z OK
2022-11-23T01:19:07.7870104Z 
2022-11-23T01:19:07.7870184Z Generating XML reports...
2022-11-23T01:19:07.7870525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011650.xml
2022-11-23T01:19:07.7870745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7871019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7871142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7871419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7871575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7871581Z 
2022-11-23T01:19:07.7871652Z Running tests...
2022-11-23T01:19:07.7871847Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7872075Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9995
2022-11-23T01:19:07.7872228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9996
2022-11-23T01:19:07.7872368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9997
2022-11-23T01:19:07.7872680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7872799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7873077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7873212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7873384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7873653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7873775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7874050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7874176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7874350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7874621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7874745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7875026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7875157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7875330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7875627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7875913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7876210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7876371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7876535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7876694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7876865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7877037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7877209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7877505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7877784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7878053Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7878347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7878587Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7878829Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7879066Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7879319Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7879600Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7879838Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7880064Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7880299Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7880534Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7880782Z STAGE:2022-11-23 01:16:56 9997:9997 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7881031Z STAGE:2022-11-23 01:16:56 9996:9996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7881099Z ok (2.273s)
2022-11-23T01:19:07.7881105Z 
2022-11-23T01:19:07.7881299Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7881370Z Ran 1 test in 2.274s
2022-11-23T01:19:07.7881376Z 
2022-11-23T01:19:07.7881432Z OK
2022-11-23T01:19:07.7881437Z 
2022-11-23T01:19:07.7881507Z Generating XML reports...
2022-11-23T01:19:07.7881846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011654.xml
2022-11-23T01:19:07.7882073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7882342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7882464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7882744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7882879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7882884Z 
2022-11-23T01:19:07.7882952Z Running tests...
2022-11-23T01:19:07.7883133Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7883371Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10056
2022-11-23T01:19:07.7883525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10057
2022-11-23T01:19:07.7883677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10058
2022-11-23T01:19:07.7883949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7884068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7884347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7884481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7884657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7884914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7885063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7885342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7885474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7885646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7885911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7886058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7886330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7886462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7886639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7886936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7887230Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7887509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7887672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7887835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7887996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7888169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7888341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7888514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7888809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7889097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7889332Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7889625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7889953Z STAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7890208Z STAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7890446Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7890701Z STAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7890955Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7891194Z STAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7891429Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7891828Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7891845Z 
2022-11-23T01:19:07.7892121Z STAGE:2022-11-23 01:17:00 10058:10058 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7892375Z STAGE:2022-11-23 01:17:00 10057:10057 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7892443Z ok (2.276s)
2022-11-23T01:19:07.7892448Z 
2022-11-23T01:19:07.7892644Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7892718Z Ran 1 test in 2.276s
2022-11-23T01:19:07.7892724Z 
2022-11-23T01:19:07.7892782Z OK
2022-11-23T01:19:07.7892788Z 
2022-11-23T01:19:07.7892869Z Generating XML reports...
2022-11-23T01:19:07.7893242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011658.xml
2022-11-23T01:19:07.7893461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7893732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7893857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7894134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7894267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7894272Z 
2022-11-23T01:19:07.7894343Z Running tests...
2022-11-23T01:19:07.7894537Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7894767Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10117
2022-11-23T01:19:07.7895028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10118
2022-11-23T01:19:07.7895172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10119
2022-11-23T01:19:07.7895445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7895565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7895843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7895976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7896150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7896421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7896542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7896819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7896941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7897115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7897409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7897678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7897798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7898078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7898212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7898385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7898669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7899010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7899174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7899337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7899498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7899672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.7899877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.7900045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.7900338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7900620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7900866Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7901103Z STAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7901513Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7901521Z 
2022-11-23T01:19:07.7901948Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7901954Z 
2022-11-23T01:19:07.7902199Z STAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7902434Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7902680Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7902919Z STAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7903172Z STAGE:2022-11-23 01:17:04 10118:10118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7903421Z STAGE:2022-11-23 01:17:04 10119:10119 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7903708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.7903772Z ok (2.179s)
2022-11-23T01:19:07.7903778Z 
2022-11-23T01:19:07.7903970Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7904044Z Ran 1 test in 2.179s
2022-11-23T01:19:07.7904050Z 
2022-11-23T01:19:07.7904107Z OK
2022-11-23T01:19:07.7904111Z 
2022-11-23T01:19:07.7904192Z Generating XML reports...
2022-11-23T01:19:07.7904533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011702.xml
2022-11-23T01:19:07.7904764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7905036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7905149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7905423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7905557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7905563Z 
2022-11-23T01:19:07.7905662Z Running tests...
2022-11-23T01:19:07.7905857Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7906082Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10178
2022-11-23T01:19:07.7906235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10179
2022-11-23T01:19:07.7906387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10180
2022-11-23T01:19:07.7906646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7906792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7907070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7907202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7907379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7907648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7907768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7908047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7908179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7908343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7908612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7908731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7909012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7909142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7909316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7909614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7909908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7910202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7910355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7910601Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7910763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7910924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7911166Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7911404Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7911649Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7912058Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7912064Z 
2022-11-23T01:19:07.7912319Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7912767Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7912774Z 
2022-11-23T01:19:07.7912999Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7913234Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7913465Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7913737Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7913991Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7914237Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7914476Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7914730Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7914978Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7915205Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7915439Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7915672Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7915913Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7916153Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7916388Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7916641Z STAGE:2022-11-23 01:17:08 10178:10178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7916889Z STAGE:2022-11-23 01:17:08 10179:10179 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7917138Z STAGE:2022-11-23 01:17:08 10180:10180 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7917194Z ok (2.276s)
2022-11-23T01:19:07.7917199Z 
2022-11-23T01:19:07.7917395Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7917469Z Ran 1 test in 2.276s
2022-11-23T01:19:07.7917475Z 
2022-11-23T01:19:07.7917532Z OK
2022-11-23T01:19:07.7917536Z 
2022-11-23T01:19:07.7917617Z Generating XML reports...
2022-11-23T01:19:07.7917960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011706.xml
2022-11-23T01:19:07.7918190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7918463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7918584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7918852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7918988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7918993Z 
2022-11-23T01:19:07.7919062Z Running tests...
2022-11-23T01:19:07.7919254Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7919508Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10234
2022-11-23T01:19:07.7919663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10235
2022-11-23T01:19:07.7919814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10236
2022-11-23T01:19:07.7920086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7920194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7920473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7920649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7920826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7921098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7921221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7921501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7921633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7921809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7922094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7922363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7922487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7922765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7922898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7923072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7923370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7923664Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7923827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7924066Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7924230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7924392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7924635Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7924872Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7925118Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7925373Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7925782Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7925790Z 
2022-11-23T01:19:07.7926217Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7926223Z 
2022-11-23T01:19:07.7926488Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7926711Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7926943Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7927186Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7927423Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7927712Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7927957Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7928211Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7928462Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7928698Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7928924Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7929157Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7929400Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7929653Z STAGE:2022-11-23 01:17:12 10234:10234 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7929985Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7930227Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7930475Z STAGE:2022-11-23 01:17:12 10235:10235 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7930724Z STAGE:2022-11-23 01:17:12 10236:10236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7930789Z ok (2.276s)
2022-11-23T01:19:07.7930795Z 
2022-11-23T01:19:07.7930978Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7931048Z Ran 1 test in 2.277s
2022-11-23T01:19:07.7931054Z 
2022-11-23T01:19:07.7931110Z OK
2022-11-23T01:19:07.7931118Z 
2022-11-23T01:19:07.7931198Z Generating XML reports...
2022-11-23T01:19:07.7931539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011710.xml
2022-11-23T01:19:07.7931769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7932047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7932169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7932437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7932569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7932574Z 
2022-11-23T01:19:07.7932644Z Running tests...
2022-11-23T01:19:07.7932838Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7933038Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T01:19:07.7933044Z 
2022-11-23T01:19:07.7933236Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7933309Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7933314Z 
2022-11-23T01:19:07.7933384Z OK (skipped=1)
2022-11-23T01:19:07.7933390Z 
2022-11-23T01:19:07.7933507Z Generating XML reports...
2022-11-23T01:19:07.7933835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011714.xml
2022-11-23T01:19:07.7934068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7934339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7934461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7934737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7935009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7935015Z 
2022-11-23T01:19:07.7935085Z Running tests...
2022-11-23T01:19:07.7935280Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7935511Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10300
2022-11-23T01:19:07.7935657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10301
2022-11-23T01:19:07.7935810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10302
2022-11-23T01:19:07.7936081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7936204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7936483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7936617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7936793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7937067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7937177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7937455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7937587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7937762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7938033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7938157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7938439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7938572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7938748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7939034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7939327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7939613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7939779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7940027Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7940189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7940394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7940641Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7940878Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7941114Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7941370Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7941813Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7941819Z 
2022-11-23T01:19:07.7942251Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7942257Z 
2022-11-23T01:19:07.7942497Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7942732Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7942965Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7943208Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7943465Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7943709Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7943961Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7944193Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7944443Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7944683Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7944920Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7945153Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7945400Z STAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7945867Z STAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:18 10300:10300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7945878Z 
2022-11-23T01:19:07.7946354Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:18 10302:10302 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7946360Z 
2022-11-23T01:19:07.7946611Z STAGE:2022-11-23 01:17:18 10301:10301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7946673Z ok (2.271s)
2022-11-23T01:19:07.7946679Z 
2022-11-23T01:19:07.7946860Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7946936Z Ran 1 test in 2.272s
2022-11-23T01:19:07.7946941Z 
2022-11-23T01:19:07.7947000Z OK
2022-11-23T01:19:07.7947005Z 
2022-11-23T01:19:07.7947087Z Generating XML reports...
2022-11-23T01:19:07.7947427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011716.xml
2022-11-23T01:19:07.7947690Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7947965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7948085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7948365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7948489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7948496Z 
2022-11-23T01:19:07.7948565Z Running tests...
2022-11-23T01:19:07.7948785Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7948999Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T01:19:07.7949005Z 
2022-11-23T01:19:07.7949195Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7949270Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7949276Z 
2022-11-23T01:19:07.7949344Z OK (skipped=1)
2022-11-23T01:19:07.7949350Z 
2022-11-23T01:19:07.7949430Z Generating XML reports...
2022-11-23T01:19:07.7949757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011720.xml
2022-11-23T01:19:07.7949985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7950260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7950384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7950664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7950798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7950804Z 
2022-11-23T01:19:07.7950876Z Running tests...
2022-11-23T01:19:07.7951067Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7951263Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-11-23T01:19:07.7951269Z 
2022-11-23T01:19:07.7951447Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7951522Z Ran 1 test in 0.004s
2022-11-23T01:19:07.7951527Z 
2022-11-23T01:19:07.7951595Z OK (skipped=1)
2022-11-23T01:19:07.7951600Z 
2022-11-23T01:19:07.7951682Z Generating XML reports...
2022-11-23T01:19:07.7952021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011722.xml
2022-11-23T01:19:07.7952251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7952521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7952644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7952922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7953043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7953048Z 
2022-11-23T01:19:07.7953117Z Running tests...
2022-11-23T01:19:07.7953307Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7953533Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10376
2022-11-23T01:19:07.7953692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10377
2022-11-23T01:19:07.7953844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10378
2022-11-23T01:19:07.7954143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7954264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7954540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7954663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7954839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7955111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7955261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7955540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7955672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7955849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7956116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7956224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7956528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7956729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7956969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7957276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7957572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7957862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7958028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7958187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7958424Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7958661Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7958908Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7959164Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7959404Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7959568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7959807Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7960053Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7960290Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7960534Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7960787Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7961023Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7961299Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7961543Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7961780Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7962013Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7962265Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7962517Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7962783Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7963023Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7963264Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7963496Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7963908Z STAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7963915Z 
2022-11-23T01:19:07.7964157Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7964585Z STAGE:2022-11-23 01:17:25 10376:10376 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:25 10377:10377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7964593Z 
2022-11-23T01:19:07.7964844Z STAGE:2022-11-23 01:17:25 10378:10378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7964909Z ok (2.272s)
2022-11-23T01:19:07.7964916Z 
2022-11-23T01:19:07.7965112Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7965175Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7965191Z 
2022-11-23T01:19:07.7965238Z OK
2022-11-23T01:19:07.7965243Z 
2022-11-23T01:19:07.7965324Z Generating XML reports...
2022-11-23T01:19:07.7965661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011723.xml
2022-11-23T01:19:07.7965892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7966166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7966289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7966567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7966703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7966708Z 
2022-11-23T01:19:07.7966767Z Running tests...
2022-11-23T01:19:07.7966960Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7967144Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:19:07.7967150Z 
2022-11-23T01:19:07.7967443Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7967516Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7967521Z 
2022-11-23T01:19:07.7967593Z OK (skipped=1)
2022-11-23T01:19:07.7967598Z 
2022-11-23T01:19:07.7967679Z Generating XML reports...
2022-11-23T01:19:07.7968016Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011727.xml
2022-11-23T01:19:07.7968244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7968539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7968666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7968947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7969080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7969085Z 
2022-11-23T01:19:07.7969156Z Running tests...
2022-11-23T01:19:07.7969346Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7969580Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:19:07.7969586Z 
2022-11-23T01:19:07.7969850Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7969912Z Ran 1 test in 0.002s
2022-11-23T01:19:07.7969929Z 
2022-11-23T01:19:07.7969988Z OK (skipped=1)
2022-11-23T01:19:07.7969995Z 
2022-11-23T01:19:07.7970075Z Generating XML reports...
2022-11-23T01:19:07.7970414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011729.xml
2022-11-23T01:19:07.7970645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7970919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7971042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7971324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7971457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7971463Z 
2022-11-23T01:19:07.7971522Z Running tests...
2022-11-23T01:19:07.7971714Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7971947Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10452
2022-11-23T01:19:07.7972102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10453
2022-11-23T01:19:07.7972257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10454
2022-11-23T01:19:07.7972529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7972649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7972934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7973067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7973232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7973506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7973628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7973907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7974040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7974214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7974487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7974606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7974966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7975186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7975364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7975666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7975962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7976253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7976452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7976697Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7976858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7977010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7977255Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7977491Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7977737Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7978146Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7978155Z 
2022-11-23T01:19:07.7978409Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7978839Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7978845Z 
2022-11-23T01:19:07.7979080Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7979316Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7979547Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7979792Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7980198Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7980215Z 
2022-11-23T01:19:07.7980453Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7980707Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7980961Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7981200Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7981433Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7981664Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7981910Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7982148Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7982421Z STAGE:2022-11-23 01:17:32 10453:10453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7982676Z STAGE:2022-11-23 01:17:32 10452:10452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7982916Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7983167Z STAGE:2022-11-23 01:17:32 10454:10454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7983233Z ok (2.273s)
2022-11-23T01:19:07.7983238Z 
2022-11-23T01:19:07.7983432Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7983532Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7983537Z 
2022-11-23T01:19:07.7983593Z OK
2022-11-23T01:19:07.7983598Z 
2022-11-23T01:19:07.7983679Z Generating XML reports...
2022-11-23T01:19:07.7984007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011730.xml
2022-11-23T01:19:07.7984239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7984513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7984636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7984914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7985044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7985052Z 
2022-11-23T01:19:07.7985123Z Running tests...
2022-11-23T01:19:07.7985313Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7985537Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10508
2022-11-23T01:19:07.7985682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10509
2022-11-23T01:19:07.7985836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10510
2022-11-23T01:19:07.7986106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7986228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7986505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7986636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7986813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.7987085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7987197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7987477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7987611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7987784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.7988054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7988179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7988534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7988725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7988984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.7989595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7989949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7990239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.7990403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.7990651Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7990846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.7991006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.7991249Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7991488Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7991724Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7991965Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7992221Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7992474Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7992716Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7992967Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7993206Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7993440Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7993671Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7993905Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7994142Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7994398Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7994823Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7994829Z 
2022-11-23T01:19:07.7995089Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7995327Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7995564Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7995799Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.7996043Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7996284Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7996513Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.7996765Z STAGE:2022-11-23 01:17:36 10508:10508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7997045Z STAGE:2022-11-23 01:17:36 10510:10510 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7997297Z STAGE:2022-11-23 01:17:36 10509:10509 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.7997362Z ok (2.273s)
2022-11-23T01:19:07.7997368Z 
2022-11-23T01:19:07.7997559Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7997632Z Ran 1 test in 2.273s
2022-11-23T01:19:07.7997638Z 
2022-11-23T01:19:07.7997695Z OK
2022-11-23T01:19:07.7997700Z 
2022-11-23T01:19:07.7997770Z Generating XML reports...
2022-11-23T01:19:07.7998110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011734.xml
2022-11-23T01:19:07.7998370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.7998646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.7998772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.7999050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.7999184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.7999189Z 
2022-11-23T01:19:07.7999259Z Running tests...
2022-11-23T01:19:07.7999453Z ----------------------------------------------------------------------
2022-11-23T01:19:07.7999672Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10564
2022-11-23T01:19:07.7999830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10565
2022-11-23T01:19:07.7999981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10566
2022-11-23T01:19:07.8000253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8000378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8000656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8000790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8000967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8001228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8001352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8001629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8001764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8001940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8002211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8002330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8002606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8002738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8002901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8003197Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8003494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8003816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8003983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8004144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8004303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8004366Z ok (2.174s)
2022-11-23T01:19:07.8004372Z 
2022-11-23T01:19:07.8004568Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8004656Z Ran 1 test in 2.174s
2022-11-23T01:19:07.8004662Z 
2022-11-23T01:19:07.8004720Z OK
2022-11-23T01:19:07.8004725Z 
2022-11-23T01:19:07.8004806Z Generating XML reports...
2022-11-23T01:19:07.8005149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011738.xml
2022-11-23T01:19:07.8005382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8005657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8005779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8006060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8006183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8006197Z 
2022-11-23T01:19:07.8006258Z Running tests...
2022-11-23T01:19:07.8006453Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8006683Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10617
2022-11-23T01:19:07.8006839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10618
2022-11-23T01:19:07.8006992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10619
2022-11-23T01:19:07.8007264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8007384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8007664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8007786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8007965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8008237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8008359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8008636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8008767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8008941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8009211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8009332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8009600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8009835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8010012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8010346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8010638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8010926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8011088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8011336Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8011526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8011674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8011921Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8012160Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8012408Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8012820Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8012827Z 
2022-11-23T01:19:07.8013085Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8013342Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8013593Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8013832Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8014058Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8014290Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8014694Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8014700Z 
2022-11-23T01:19:07.8015052Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8015486Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8015492Z 
2022-11-23T01:19:07.8015746Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8016139Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8016145Z 
2022-11-23T01:19:07.8016380Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8016623Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8016875Z STAGE:2022-11-23 01:17:44 10617:10617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8017287Z STAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8017293Z 
2022-11-23T01:19:07.8017767Z STAGE:2022-11-23 01:17:44 10618:10618 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:44 10619:10619 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8017773Z 
2022-11-23T01:19:07.8017840Z ok (2.271s)
2022-11-23T01:19:07.8017845Z 
2022-11-23T01:19:07.8018032Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8018147Z Ran 1 test in 2.272s
2022-11-23T01:19:07.8018152Z 
2022-11-23T01:19:07.8018212Z OK
2022-11-23T01:19:07.8018216Z 
2022-11-23T01:19:07.8018298Z Generating XML reports...
2022-11-23T01:19:07.8018642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011742.xml
2022-11-23T01:19:07.8018925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8019202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8019324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8019605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8019730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8019735Z 
2022-11-23T01:19:07.8019805Z Running tests...
2022-11-23T01:19:07.8019998Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8020179Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-11-23T01:19:07.8020187Z 
2022-11-23T01:19:07.8020375Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8020449Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8020454Z 
2022-11-23T01:19:07.8020523Z OK (skipped=1)
2022-11-23T01:19:07.8020528Z 
2022-11-23T01:19:07.8020609Z Generating XML reports...
2022-11-23T01:19:07.8020938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011746.xml
2022-11-23T01:19:07.8021170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8021442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8021563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8021843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8021978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8021984Z 
2022-11-23T01:19:07.8022053Z Running tests...
2022-11-23T01:19:07.8022243Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8022436Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:19:07.8022442Z 
2022-11-23T01:19:07.8022622Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8022695Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8022700Z 
2022-11-23T01:19:07.8022768Z OK (skipped=1)
2022-11-23T01:19:07.8022772Z 
2022-11-23T01:19:07.8022856Z Generating XML reports...
2022-11-23T01:19:07.8023194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011748.xml
2022-11-23T01:19:07.8023424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8023698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8023819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8024094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8024247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8024253Z 
2022-11-23T01:19:07.8024322Z Running tests...
2022-11-23T01:19:07.8024510Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8024739Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10693
2022-11-23T01:19:07.8024890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10694
2022-11-23T01:19:07.8025041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10695
2022-11-23T01:19:07.8025343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8025464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8025731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8025865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8026040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8026310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8026431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8026706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8026840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8027017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8027284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8027396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8027673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8027806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8027980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8028279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8028575Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8028866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8029031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8029197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8029345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8029518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.8029690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.8029861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.8030157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8030445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8030761Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8031012Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8031414Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8031420Z 
2022-11-23T01:19:07.8031826Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8031860Z 
2022-11-23T01:19:07.8032106Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8032526Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8032543Z 
2022-11-23T01:19:07.8032784Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8033173Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8033179Z 
2022-11-23T01:19:07.8033415Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8033662Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8033902Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8034138Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8034393Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8034642Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8034892Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8035120Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8035353Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8035587Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8035831Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8036069Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8036309Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8036559Z STAGE:2022-11-23 01:17:51 10693:10693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8036808Z STAGE:2022-11-23 01:17:51 10695:10695 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8037055Z STAGE:2022-11-23 01:17:51 10694:10694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8037109Z ok (2.271s)
2022-11-23T01:19:07.8037114Z 
2022-11-23T01:19:07.8037311Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8037385Z Ran 1 test in 2.271s
2022-11-23T01:19:07.8037390Z 
2022-11-23T01:19:07.8037449Z OK
2022-11-23T01:19:07.8037454Z 
2022-11-23T01:19:07.8037537Z Generating XML reports...
2022-11-23T01:19:07.8037911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011750.xml
2022-11-23T01:19:07.8038143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8038415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8038538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8038806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8038939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8038972Z 
2022-11-23T01:19:07.8039044Z Running tests...
2022-11-23T01:19:07.8039238Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8039466Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10758
2022-11-23T01:19:07.8039622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10759
2022-11-23T01:19:07.8039777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10760
2022-11-23T01:19:07.8040046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8040154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8040434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8040566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8040746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8041015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8041138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8041420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8041553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8041728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8041986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8042107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8042385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8042518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8042694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8042995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8043290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8043583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8043747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8043898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8044061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8044233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:19:07.8044433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:19:07.8044603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:19:07.8044898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8045187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8045434Z STAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8045753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:19:07.8045987Z STAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8046404Z STAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8046411Z 
2022-11-23T01:19:07.8046840Z STAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8046846Z 
2022-11-23T01:19:07.8047242Z STAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8047250Z 
2022-11-23T01:19:07.8047495Z STAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8047737Z STAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8047990Z STAGE:2022-11-23 01:17:55 10760:10760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8048246Z STAGE:2022-11-23 01:17:55 10759:10759 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8048312Z ok (2.274s)
2022-11-23T01:19:07.8048318Z 
2022-11-23T01:19:07.8048506Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8053839Z Ran 1 test in 2.274s
2022-11-23T01:19:07.8053849Z 
2022-11-23T01:19:07.8053918Z OK
2022-11-23T01:19:07.8053923Z 
2022-11-23T01:19:07.8054016Z Generating XML reports...
2022-11-23T01:19:07.8054387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011754.xml
2022-11-23T01:19:07.8054627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8055062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8055189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8055464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8055601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8055606Z 
2022-11-23T01:19:07.8055681Z Running tests...
2022-11-23T01:19:07.8055880Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8056112Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10819
2022-11-23T01:19:07.8056269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10820
2022-11-23T01:19:07.8056420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10821
2022-11-23T01:19:07.8056695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8056901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8057188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8057323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8057503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8057773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8057894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8058217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8058349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8058522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8058783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8058902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8059181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8059314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8059487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8059785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8060080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8060373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8060537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8060688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8060850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8060916Z ok (2.174s)
2022-11-23T01:19:07.8060923Z 
2022-11-23T01:19:07.8061120Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8061194Z Ran 1 test in 2.174s
2022-11-23T01:19:07.8061200Z 
2022-11-23T01:19:07.8061257Z OK
2022-11-23T01:19:07.8061263Z 
2022-11-23T01:19:07.8061343Z Generating XML reports...
2022-11-23T01:19:07.8061688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011758.xml
2022-11-23T01:19:07.8061909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8062182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8062301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8062578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8062713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8062719Z 
2022-11-23T01:19:07.8062786Z Running tests...
2022-11-23T01:19:07.8062980Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8063203Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10872
2022-11-23T01:19:07.8063357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10873
2022-11-23T01:19:07.8063528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10874
2022-11-23T01:19:07.8063803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8063925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8064210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8064342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8064518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8064848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8065119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8065241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8065508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8065638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8065812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8066081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8066200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8066478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8066609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8066782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8067067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8067360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8067521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8067682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8067840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8067908Z ok (2.273s)
2022-11-23T01:19:07.8067914Z 
2022-11-23T01:19:07.8068107Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8068176Z Ran 1 test in 2.273s
2022-11-23T01:19:07.8068181Z 
2022-11-23T01:19:07.8068239Z OK
2022-11-23T01:19:07.8068244Z 
2022-11-23T01:19:07.8068317Z Generating XML reports...
2022-11-23T01:19:07.8068659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011801.xml
2022-11-23T01:19:07.8068890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8069162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8069282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8069562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8069698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8069703Z 
2022-11-23T01:19:07.8069772Z Running tests...
2022-11-23T01:19:07.8069954Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8070212Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10925
2022-11-23T01:19:07.8070370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10926
2022-11-23T01:19:07.8070521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10927
2022-11-23T01:19:07.8070793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8070910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8071188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8071365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8071537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8071802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8071922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8072204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8072336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8072509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8072775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8072896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8073172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8073301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8073466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8073760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8074056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8074346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8074512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8074675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8074834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8074897Z ok (2.074s)
2022-11-23T01:19:07.8074902Z 
2022-11-23T01:19:07.8075088Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8075157Z Ran 1 test in 2.074s
2022-11-23T01:19:07.8075163Z 
2022-11-23T01:19:07.8075219Z OK
2022-11-23T01:19:07.8075224Z 
2022-11-23T01:19:07.8075307Z Generating XML reports...
2022-11-23T01:19:07.8075647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011805.xml
2022-11-23T01:19:07.8075878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8076147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8076270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8076545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8076694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8076700Z 
2022-11-23T01:19:07.8076771Z Running tests...
2022-11-23T01:19:07.8076963Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8077216Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10978
2022-11-23T01:19:07.8077372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10979
2022-11-23T01:19:07.8077522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10980
2022-11-23T01:19:07.8077821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8077940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8078217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8078341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8078514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8078782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8078901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8079180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8079313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8079487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8079756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8079868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8080146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8080279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8080451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8080749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8081040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8081330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8081495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8081656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8081893Z STAGE:2022-11-23 01:18:11 10980:10980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8082131Z STAGE:2022-11-23 01:18:11 10979:10979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8082293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8082534Z STAGE:2022-11-23 01:18:11 10978:10978 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8082784Z STAGE:2022-11-23 01:18:11 10978:10978 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8083022Z STAGE:2022-11-23 01:18:11 10980:10980 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8083277Z STAGE:2022-11-23 01:18:11 10978:10978 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8083548Z STAGE:2022-11-23 01:18:11 10979:10979 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8083806Z STAGE:2022-11-23 01:18:11 10980:10980 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8084046Z STAGE:2022-11-23 01:18:11 10979:10979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8084110Z ok (2.273s)
2022-11-23T01:19:07.8084116Z 
2022-11-23T01:19:07.8084309Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8084381Z Ran 1 test in 2.273s
2022-11-23T01:19:07.8084447Z 
2022-11-23T01:19:07.8084504Z OK
2022-11-23T01:19:07.8084509Z 
2022-11-23T01:19:07.8084591Z Generating XML reports...
2022-11-23T01:19:07.8084936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011809.xml
2022-11-23T01:19:07.8085166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8085441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8085552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8085832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8085965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8085971Z 
2022-11-23T01:19:07.8086042Z Running tests...
2022-11-23T01:19:07.8086236Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8086487Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11034
2022-11-23T01:19:07.8086644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11035
2022-11-23T01:19:07.8086800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11036
2022-11-23T01:19:07.8087059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8087179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8087462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8087595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8087772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8088045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8088164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8088447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8088577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8088740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8089007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8089125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8089404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8089537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8089784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8090086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8090418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8090710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8090860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8091112Z STAGE:2022-11-23 01:18:15 11036:11036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8091271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8091460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8091702Z STAGE:2022-11-23 01:18:15 11034:11034 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8091940Z STAGE:2022-11-23 01:18:15 11035:11035 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8092185Z STAGE:2022-11-23 01:18:15 11034:11034 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8092426Z STAGE:2022-11-23 01:18:15 11035:11035 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8092678Z STAGE:2022-11-23 01:18:15 11034:11034 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8092910Z STAGE:2022-11-23 01:18:15 11036:11036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8093162Z STAGE:2022-11-23 01:18:15 11035:11035 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8093413Z STAGE:2022-11-23 01:18:15 11036:11036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8093479Z ok (2.275s)
2022-11-23T01:19:07.8093485Z 
2022-11-23T01:19:07.8093676Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8093751Z Ran 1 test in 2.275s
2022-11-23T01:19:07.8093758Z 
2022-11-23T01:19:07.8093813Z OK
2022-11-23T01:19:07.8093818Z 
2022-11-23T01:19:07.8093899Z Generating XML reports...
2022-11-23T01:19:07.8094226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011813.xml
2022-11-23T01:19:07.8094460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8094732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8094969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8095253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8095388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8095394Z 
2022-11-23T01:19:07.8095463Z Running tests...
2022-11-23T01:19:07.8095658Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8095901Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11090
2022-11-23T01:19:07.8096047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11091
2022-11-23T01:19:07.8096201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11092
2022-11-23T01:19:07.8096474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8096596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8096875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8097006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8097229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8097526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8097794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8097905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8098183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8098353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8098527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8098796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8098919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8099195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8099328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8099490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8099783Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8100077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8100240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8100401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8100653Z STAGE:2022-11-23 01:18:19 11090:11090 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8100812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8101055Z STAGE:2022-11-23 01:18:19 11092:11092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8101292Z STAGE:2022-11-23 01:18:19 11091:11091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8101527Z STAGE:2022-11-23 01:18:19 11092:11092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8101785Z STAGE:2022-11-23 01:18:19 11092:11092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8102028Z STAGE:2022-11-23 01:18:19 11090:11090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8102455Z STAGE:2022-11-23 01:18:19 11091:11091 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:18:19 11090:11090 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8102462Z 
2022-11-23T01:19:07.8102717Z STAGE:2022-11-23 01:18:19 11091:11091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8102783Z ok (2.073s)
2022-11-23T01:19:07.8102789Z 
2022-11-23T01:19:07.8102985Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8103056Z Ran 1 test in 2.074s
2022-11-23T01:19:07.8103061Z 
2022-11-23T01:19:07.8103119Z OK
2022-11-23T01:19:07.8103124Z 
2022-11-23T01:19:07.8103194Z Generating XML reports...
2022-11-23T01:19:07.8103538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011817.xml
2022-11-23T01:19:07.8103766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8104065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8104187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8104464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8104595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8104601Z 
2022-11-23T01:19:07.8104669Z Running tests...
2022-11-23T01:19:07.8104868Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8105025Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-11-23T01:19:07.8105056Z 
2022-11-23T01:19:07.8105246Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8105317Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8105323Z 
2022-11-23T01:19:07.8105394Z OK (skipped=1)
2022-11-23T01:19:07.8105399Z 
2022-11-23T01:19:07.8105480Z Generating XML reports...
2022-11-23T01:19:07.8105816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011821.xml
2022-11-23T01:19:07.8106044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8106315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8106435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8106701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8106835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8106841Z 
2022-11-23T01:19:07.8106911Z Running tests...
2022-11-23T01:19:07.8107101Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8107293Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-11-23T01:19:07.8107298Z 
2022-11-23T01:19:07.8107487Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8107557Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8107562Z 
2022-11-23T01:19:07.8107627Z OK (skipped=1)
2022-11-23T01:19:07.8107632Z 
2022-11-23T01:19:07.8107710Z Generating XML reports...
2022-11-23T01:19:07.8108035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011823.xml
2022-11-23T01:19:07.8108262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8108537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8108656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8108934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8109068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8109074Z 
2022-11-23T01:19:07.8109144Z Running tests...
2022-11-23T01:19:07.8109336Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8109511Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:19:07.8109524Z 
2022-11-23T01:19:07.8109701Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8109770Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8109778Z 
2022-11-23T01:19:07.8109848Z OK (skipped=1)
2022-11-23T01:19:07.8109853Z 
2022-11-23T01:19:07.8109935Z Generating XML reports...
2022-11-23T01:19:07.8110269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011824.xml
2022-11-23T01:19:07.8110531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8110800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8110917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8111184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8111315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8111320Z 
2022-11-23T01:19:07.8111390Z Running tests...
2022-11-23T01:19:07.8111611Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8111846Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11176
2022-11-23T01:19:07.8111999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11177
2022-11-23T01:19:07.8112153Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11178
2022-11-23T01:19:07.8112422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8112543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8112811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8112941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8113116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8113418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8113686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8113808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8114085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8114217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8114381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8114647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8114765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8115043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8115173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8115344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8115644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8115937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8116101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8116249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8116495Z STAGE:2022-11-23 01:18:28 11177:11177 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8116735Z STAGE:2022-11-23 01:18:28 11178:11178 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8116898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8117167Z STAGE:2022-11-23 01:18:28 11176:11176 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8117413Z STAGE:2022-11-23 01:18:28 11178:11178 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8117669Z STAGE:2022-11-23 01:18:28 11178:11178 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8117914Z STAGE:2022-11-23 01:18:28 11176:11176 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8118152Z STAGE:2022-11-23 01:18:28 11177:11177 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8118395Z STAGE:2022-11-23 01:18:28 11176:11176 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8118686Z STAGE:2022-11-23 01:18:28 11177:11177 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8118751Z ok (2.272s)
2022-11-23T01:19:07.8118756Z 
2022-11-23T01:19:07.8118951Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8119026Z Ran 1 test in 2.272s
2022-11-23T01:19:07.8119032Z 
2022-11-23T01:19:07.8119089Z OK
2022-11-23T01:19:07.8119094Z 
2022-11-23T01:19:07.8119175Z Generating XML reports...
2022-11-23T01:19:07.8119513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011826.xml
2022-11-23T01:19:07.8119731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8120000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8120122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8120398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8120530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8120535Z 
2022-11-23T01:19:07.8120605Z Running tests...
2022-11-23T01:19:07.8120798Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8121026Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11232
2022-11-23T01:19:07.8121176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11233
2022-11-23T01:19:07.8121318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11234
2022-11-23T01:19:07.8121588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8121712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8121988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8122119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8122294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8122562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8122684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8122963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8123084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8123255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8123527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8123647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8123949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8124082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8124255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8124553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8124836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8125157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8125320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8125480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8125641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8125706Z ok (2.074s)
2022-11-23T01:19:07.8125712Z 
2022-11-23T01:19:07.8125905Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8125978Z Ran 1 test in 2.074s
2022-11-23T01:19:07.8125983Z 
2022-11-23T01:19:07.8126039Z OK
2022-11-23T01:19:07.8126044Z 
2022-11-23T01:19:07.8126114Z Generating XML reports...
2022-11-23T01:19:07.8126452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011830.xml
2022-11-23T01:19:07.8126683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8126951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8127072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8127348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8127481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8127487Z 
2022-11-23T01:19:07.8127556Z Running tests...
2022-11-23T01:19:07.8127748Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8127985Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11285
2022-11-23T01:19:07.8128138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11286
2022-11-23T01:19:07.8128289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11287
2022-11-23T01:19:07.8128558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8128678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8128958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8129088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8129261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8129521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8129639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8129997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8130134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8130307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8130638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8130909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8131030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8131307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8131427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8131627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8131923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8132221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8132383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8132544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8132791Z STAGE:2022-11-23 01:18:35 11285:11285 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8133031Z STAGE:2022-11-23 01:18:35 11287:11287 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8133192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8133425Z STAGE:2022-11-23 01:18:35 11286:11286 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8133670Z STAGE:2022-11-23 01:18:35 11285:11285 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8133924Z STAGE:2022-11-23 01:18:35 11285:11285 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8134170Z STAGE:2022-11-23 01:18:35 11287:11287 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8134413Z STAGE:2022-11-23 01:18:35 11286:11286 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8134668Z STAGE:2022-11-23 01:18:35 11287:11287 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8135031Z STAGE:2022-11-23 01:18:35 11286:11286 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8135099Z ok (2.174s)
2022-11-23T01:19:07.8135105Z 
2022-11-23T01:19:07.8135305Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8135366Z Ran 1 test in 2.174s
2022-11-23T01:19:07.8135371Z 
2022-11-23T01:19:07.8135428Z OK
2022-11-23T01:19:07.8135433Z 
2022-11-23T01:19:07.8135513Z Generating XML reports...
2022-11-23T01:19:07.8135859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011834.xml
2022-11-23T01:19:07.8136092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8136366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8136487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8136764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8136887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8136906Z 
2022-11-23T01:19:07.8136966Z Running tests...
2022-11-23T01:19:07.8137157Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8137401Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11341
2022-11-23T01:19:07.8137606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11342
2022-11-23T01:19:07.8137761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11343
2022-11-23T01:19:07.8138032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8138152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8138433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8138592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8138769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8139042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8139165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8139440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8139574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8139748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8140020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8140129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8140407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8140541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8140712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8141010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8141303Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8141593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8141756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8141921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8142157Z STAGE:2022-11-23 01:18:39 11342:11342 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8142396Z STAGE:2022-11-23 01:18:39 11343:11343 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8142557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8142799Z STAGE:2022-11-23 01:18:39 11341:11341 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:19:07.8143046Z STAGE:2022-11-23 01:18:39 11341:11341 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8143301Z STAGE:2022-11-23 01:18:39 11341:11341 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8143545Z STAGE:2022-11-23 01:18:39 11342:11342 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:19:07.8143966Z STAGE:2022-11-23 01:18:39 11343:11343 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:18:39 11342:11342 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8143974Z 
2022-11-23T01:19:07.8144227Z STAGE:2022-11-23 01:18:39 11343:11343 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:19:07.8144291Z ok (2.174s)
2022-11-23T01:19:07.8144345Z 
2022-11-23T01:19:07.8144531Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8144603Z Ran 1 test in 2.174s
2022-11-23T01:19:07.8144609Z 
2022-11-23T01:19:07.8144666Z OK
2022-11-23T01:19:07.8144671Z 
2022-11-23T01:19:07.8144750Z Generating XML reports...
2022-11-23T01:19:07.8145093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011838.xml
2022-11-23T01:19:07.8145324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8145631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8145752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8146034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8146157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8146162Z 
2022-11-23T01:19:07.8146232Z Running tests...
2022-11-23T01:19:07.8146427Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8146662Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11397
2022-11-23T01:19:07.8146816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11398
2022-11-23T01:19:07.8146971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11399
2022-11-23T01:19:07.8147244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8147360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8147627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8147758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8147933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8148202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8148320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8148600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8148734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8148909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8149176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8149286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8149561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8149696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8149869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8150168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8150467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8150759Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8150922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8151112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8151262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8151328Z ok (2.272s)
2022-11-23T01:19:07.8151333Z 
2022-11-23T01:19:07.8151524Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8151596Z Ran 1 test in 2.273s
2022-11-23T01:19:07.8151602Z 
2022-11-23T01:19:07.8151659Z OK
2022-11-23T01:19:07.8151664Z 
2022-11-23T01:19:07.8151744Z Generating XML reports...
2022-11-23T01:19:07.8152108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011842.xml
2022-11-23T01:19:07.8152338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8152596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8152717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8152995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8153131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8153136Z 
2022-11-23T01:19:07.8153205Z Running tests...
2022-11-23T01:19:07.8153396Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8153637Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11477
2022-11-23T01:19:07.8153791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11478
2022-11-23T01:19:07.8153941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11479
2022-11-23T01:19:07.8154200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8154318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8154597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8154730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8154903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8155198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8155469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8155590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8155871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8155993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8156166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8156435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8156556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8156832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8156963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8157132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8157420Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8157735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8157897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8158058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8158218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8158310Z skip: CUDA is not available. (2.172s)
2022-11-23T01:19:07.8158316Z 
2022-11-23T01:19:07.8158533Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8158605Z Ran 1 test in 2.173s
2022-11-23T01:19:07.8158610Z 
2022-11-23T01:19:07.8158677Z OK (skipped=1)
2022-11-23T01:19:07.8158682Z 
2022-11-23T01:19:07.8158764Z Generating XML reports...
2022-11-23T01:19:07.8159094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011846.xml
2022-11-23T01:19:07.8159321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8159592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8159713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8159993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8160124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8160131Z 
2022-11-23T01:19:07.8160199Z Running tests...
2022-11-23T01:19:07.8160389Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8160628Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11530
2022-11-23T01:19:07.8160772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11531
2022-11-23T01:19:07.8160924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11532
2022-11-23T01:19:07.8161195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8161314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8161592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8161725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8161900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8162170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8162280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8162558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8162688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8162862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8163133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8163251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8163531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8163662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8163833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8164147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8164443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8164730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8164894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8165094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8165254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8165355Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:19:07.8165361Z 
2022-11-23T01:19:07.8165554Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8165626Z Ran 1 test in 2.176s
2022-11-23T01:19:07.8165632Z 
2022-11-23T01:19:07.8165692Z OK (skipped=1)
2022-11-23T01:19:07.8165697Z 
2022-11-23T01:19:07.8165776Z Generating XML reports...
2022-11-23T01:19:07.8166117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011849.xml
2022-11-23T01:19:07.8166344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8166619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8166741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8167020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8167153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8167158Z 
2022-11-23T01:19:07.8167218Z Running tests...
2022-11-23T01:19:07.8167407Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8167642Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11583
2022-11-23T01:19:07.8167795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11584
2022-11-23T01:19:07.8167949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11585
2022-11-23T01:19:07.8168220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8168340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8168618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8168747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8168913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8169209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8169477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8169597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8169953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8170089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8170263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8170534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8170687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8170951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8171082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8171254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8171554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8171879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8172044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8172206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8172391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_e7zoeb
2022-11-23T01:19:07.8172574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_e7zoeb/_remote_module_non_scriptable.py
2022-11-23T01:19:07.8172756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8sna4iz
2022-11-23T01:19:07.8172948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8sna4iz/_remote_module_non_scriptable.py
2022-11-23T01:19:07.8173108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8173293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaub2z3k7
2022-11-23T01:19:07.8173482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaub2z3k7/_remote_module_non_scriptable.py
2022-11-23T01:19:07.8173549Z ok (2.174s)
2022-11-23T01:19:07.8173554Z 
2022-11-23T01:19:07.8173752Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8173825Z Ran 1 test in 2.175s
2022-11-23T01:19:07.8173831Z 
2022-11-23T01:19:07.8173877Z OK
2022-11-23T01:19:07.8173882Z 
2022-11-23T01:19:07.8173962Z Generating XML reports...
2022-11-23T01:19:07.8174301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011853.xml
2022-11-23T01:19:07.8174532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8174801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8175051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8175332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8175468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8175474Z 
2022-11-23T01:19:07.8175545Z Running tests...
2022-11-23T01:19:07.8175727Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8175953Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11636
2022-11-23T01:19:07.8176108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11637
2022-11-23T01:19:07.8176260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11638
2022-11-23T01:19:07.8176531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8176654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8176935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8177066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8177278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8177548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8177667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8177947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8178077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8178285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8178560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8178681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8178960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8179082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8179255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8179553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8179848Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8180145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8180310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8180470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8180632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8180732Z skip: Need at least 2 CUDA devices (2.072s)
2022-11-23T01:19:07.8180738Z 
2022-11-23T01:19:07.8180919Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8180990Z Ran 1 test in 2.073s
2022-11-23T01:19:07.8180995Z 
2022-11-23T01:19:07.8181064Z OK (skipped=1)
2022-11-23T01:19:07.8181069Z 
2022-11-23T01:19:07.8181151Z Generating XML reports...
2022-11-23T01:19:07.8181490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011857.xml
2022-11-23T01:19:07.8181720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8181989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8182112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8182378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8182509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8182515Z 
2022-11-23T01:19:07.8182581Z Running tests...
2022-11-23T01:19:07.8182773Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8183029Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11689
2022-11-23T01:19:07.8183187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11690
2022-11-23T01:19:07.8183339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11691
2022-11-23T01:19:07.8183608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8183757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8184029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8184157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8184330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:19:07.8184602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8184749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8185026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8185157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8185331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:07.8185600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8185709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8185982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8186113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8186283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:07.8186581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8186875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8187165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:19:07.8187329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:19:07.8187483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:07.8187640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:07.8187739Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:19:07.8187745Z 
2022-11-23T01:19:07.8187937Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8188010Z Ran 1 test in 2.174s
2022-11-23T01:19:07.8188015Z 
2022-11-23T01:19:07.8188081Z OK (skipped=1)
2022-11-23T01:19:07.8188087Z 
2022-11-23T01:19:07.8188167Z Generating XML reports...
2022-11-23T01:19:07.8188509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011901.xml
2022-11-23T01:19:07.8188738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8188998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8189118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8189399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8189531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8189538Z 
2022-11-23T01:19:07.8189607Z Running tests...
2022-11-23T01:19:07.8189798Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8190144Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:19:07.8190150Z 
2022-11-23T01:19:07.8190402Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8190475Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8190481Z 
2022-11-23T01:19:07.8190537Z OK (skipped=1)
2022-11-23T01:19:07.8190542Z 
2022-11-23T01:19:07.8190622Z Generating XML reports...
2022-11-23T01:19:07.8190959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011905.xml
2022-11-23T01:19:07.8191190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:19:07.8191460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:07.8191609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:07.8191887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:07.8192026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:07.8192032Z 
2022-11-23T01:19:07.8192100Z Running tests...
2022-11-23T01:19:07.8192280Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8192632Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.001s)
2022-11-23T01:19:07.8192638Z 
2022-11-23T01:19:07.8192828Z ----------------------------------------------------------------------
2022-11-23T01:19:07.8192900Z Ran 1 test in 0.002s
2022-11-23T01:19:07.8192905Z 
2022-11-23T01:19:07.8192976Z OK (skipped=1)
2022-11-23T01:19:07.8192980Z 
2022-11-23T01:19:07.8193060Z Generating XML reports...
2022-11-23T01:19:07.8193398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011907.xml
2022-11-23T01:19:07.8193403Z 
2022-11-23T01:19:07.8193762Z ##[endgroup]
2022-11-23T01:19:07.8194117Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fe0m1lpv)
2022-11-23T01:19:07.8194123Z 
2022-11-23T01:19:07.8194269Z Running distributed tests for the gloo backend with file init_method in shard 1 of 2
2022-11-23T01:19:07.8194663Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:07.571774]
2022-11-23T01:33:43.4618166Z 
2022-11-23T01:33:43.4618557Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:33:43.4620857Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_1yy7fq2y)
2022-11-23T01:33:43.4627865Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:33:43.4671962Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:33:43.4710259Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4735775Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4737994Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4738474Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4739019Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4739437Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4740066Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4740528Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4741007Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4741614Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4742116Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4742705Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4743315Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4743973Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4744510Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4745017Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4745549Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4746033Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4746494Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4746994Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4747709Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4748257Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4748715Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4749164Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4749652Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4750110Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4750661Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4751148Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4751626Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4752064Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4752505Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4752961Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4753414Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4753853Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4754335Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4754803Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4755267Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4755756Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4756227Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4756698Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4757169Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4757673Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4758143Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4758668Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4759157Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4759581Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4760060Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4760560Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4761043Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4761535Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4762040Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4762529Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4762975Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4763449Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4763762Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4764176Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4764639Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4765145Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4765596Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4765914Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4766343Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4766914Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4767327Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4767810Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4768249Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4768706Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4769157Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4769604Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4770054Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4770435Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4770869Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4771317Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4771745Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4772229Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4772658Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4773080Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4773422Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4773818Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4774277Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4774635Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4775222Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4775628Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4776028Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4776502Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4776991Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4777493Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4777969Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4778461Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4778944Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4779428Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4779951Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4780472Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4780967Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4781481Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4781989Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4782502Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4783058Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4783605Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4784616Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4785241Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4785664Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4786090Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4786514Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4786992Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4787431Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4787844Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4788294Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4788780Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4789237Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4789708Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4790132Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4790657Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4791105Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4791562Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4792015Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4792506Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4792982Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4793371Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4793772Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4794075Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4794345Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4794620Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4794904Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4795176Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4795457Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4795746Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4796080Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4796447Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4796779Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4797081Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4797376Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4797696Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4798025Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4798370Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4798682Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4798993Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4799315Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4799602Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4799874Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4800152Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4816304Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4816766Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4817383Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4817742Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4818050Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4818622Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4819217Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4819592Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4820179Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4820962Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4821943Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4822727Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4823319Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4823819Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4824462Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4825248Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4825925Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4826553Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4827166Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4827691Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4828235Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4828713Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4829229Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4829726Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4830238Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4830872Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4831512Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4832037Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4832566Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4833123Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4833647Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4834205Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4834773Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4835337Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4835869Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4836425Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4837009Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4837564Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4838101Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4838657Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4839257Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4839555Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4839896Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4840220Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4840513Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4840779Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4841107Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4841415Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4841705Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4841984Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4842253Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4842509Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4842789Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4843068Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4843340Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4843609Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4843891Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4844155Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4844410Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4844686Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4844975Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4845249Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4845521Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4845777Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4846043Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4846339Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4846643Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4846977Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4847298Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4847604Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4847996Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4848305Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4848617Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4848932Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4849237Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4849528Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4849831Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4850132Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4850408Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4850701Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4851045Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4851397Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4851742Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4852107Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4852450Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4852769Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4853087Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4853397Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4853699Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4854055Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4854384Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4854741Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4855270Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4855834Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4856362Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4856885Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4857427Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4857926Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4858387Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4858840Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4859316Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4859785Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4860202Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4860639Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4861086Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4861548Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4862029Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4862525Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4862971Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4863399Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4863866Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4864327Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4864754Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4865201Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4865638Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4866087Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4866539Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4867010Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4867476Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4867912Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4868365Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4868818Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4869345Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4869869Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4870402Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4870890Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4871473Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4872015Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4872518Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4873001Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4873495Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4874029Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4874547Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4875086Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4875575Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4876057Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4876493Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4877005Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4877559Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4878103Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.4878900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4879697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4880258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4880968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4881529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4881801Z 
2022-11-23T01:33:43.4881921Z Running tests...
2022-11-23T01:33:43.4882435Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4883174Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11772
2022-11-23T01:33:43.4883937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11773
2022-11-23T01:33:43.4884485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11774
2022-11-23T01:33:43.4885231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4885760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4886483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4887057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4887867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4888639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4889180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4889892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4890451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4890994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4891814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4892635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4893241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4893928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4894493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4896284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4897100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4898052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4898690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4899262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4899831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4900313Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:33:43.4900538Z 
2022-11-23T01:33:43.4900871Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4901254Z Ran 1 test in 2.075s
2022-11-23T01:33:43.4901424Z 
2022-11-23T01:33:43.4901552Z OK (skipped=1)
2022-11-23T01:33:43.4901720Z 
2022-11-23T01:33:43.4901864Z Generating XML reports...
2022-11-23T01:33:43.4902641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011910.xml
2022-11-23T01:33:43.4903464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4904386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4904939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4905653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4906209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4906465Z 
2022-11-23T01:33:43.4906587Z Running tests...
2022-11-23T01:33:43.4907088Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4907699Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-11-23T01:33:43.4908053Z 
2022-11-23T01:33:43.4908371Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4908760Z Ran 1 test in 0.003s
2022-11-23T01:33:43.4908937Z 
2022-11-23T01:33:43.4909051Z OK (skipped=1)
2022-11-23T01:33:43.4909223Z 
2022-11-23T01:33:43.4909334Z Generating XML reports...
2022-11-23T01:33:43.4910104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011914.xml
2022-11-23T01:33:43.4910909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4911686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4912206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4913004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4913676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4913976Z 
2022-11-23T01:33:43.4914106Z Running tests...
2022-11-23T01:33:43.4914620Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4915241Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11835
2022-11-23T01:33:43.4915716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11836
2022-11-23T01:33:43.4916037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11837
2022-11-23T01:33:43.4916489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4916823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4917251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4917618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4917950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4918419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4918737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4919165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4919509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4919843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4920303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4920633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4921058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4921398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4921714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4922208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4922728Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4923230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4923611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4923961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4924304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4924543Z ok (2.176s)
2022-11-23T01:33:43.4924646Z 
2022-11-23T01:33:43.4924842Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4925078Z Ran 1 test in 2.176s
2022-11-23T01:33:43.4925190Z 
2022-11-23T01:33:43.4925237Z OK
2022-11-23T01:33:43.4925327Z 
2022-11-23T01:33:43.4925413Z Generating XML reports...
2022-11-23T01:33:43.4925869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011915.xml
2022-11-23T01:33:43.4926365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4926812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4927142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4927567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4928000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4928150Z 
2022-11-23T01:33:43.4928223Z Running tests...
2022-11-23T01:33:43.4928559Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4929441Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:33:43.4929911Z 
2022-11-23T01:33:43.4930107Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4930365Z Ran 1 test in 0.254s
2022-11-23T01:33:43.4930479Z 
2022-11-23T01:33:43.4930552Z OK (skipped=1)
2022-11-23T01:33:43.4930660Z 
2022-11-23T01:33:43.4930744Z Generating XML reports...
2022-11-23T01:33:43.4931199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011919.xml
2022-11-23T01:33:43.4931680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4932140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4932471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4932884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4933229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4933394Z 
2022-11-23T01:33:43.4933466Z Running tests...
2022-11-23T01:33:43.4933764Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4934172Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11898
2022-11-23T01:33:43.4934595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11899
2022-11-23T01:33:43.4935098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11900
2022-11-23T01:33:43.4935574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4935892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4936322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4936667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4936993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4937458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4937789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4938220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4938551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4938885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4939350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4939664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4940090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4940430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4940759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4941294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4941818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4942337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4942720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4943055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4943475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbrlntvj
2022-11-23T01:33:43.4943864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0oa2hoaf
2022-11-23T01:33:43.4944249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbrlntvj/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4944657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0oa2hoaf/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4945031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4945396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusde1z2t
2022-11-23T01:33:43.4945779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusde1z2t/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4946159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4946516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4946874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4947115Z ok (2.273s)
2022-11-23T01:33:43.4947215Z 
2022-11-23T01:33:43.4947414Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4947656Z Ran 1 test in 2.273s
2022-11-23T01:33:43.4947768Z 
2022-11-23T01:33:43.4947816Z OK
2022-11-23T01:33:43.4947905Z 
2022-11-23T01:33:43.4947989Z Generating XML reports...
2022-11-23T01:33:43.4948447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011921.xml
2022-11-23T01:33:43.4948942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4949389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4949721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4950146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4950477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4950638Z 
2022-11-23T01:33:43.4950709Z Running tests...
2022-11-23T01:33:43.4951011Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4951444Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11951
2022-11-23T01:33:43.4951863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11952
2022-11-23T01:33:43.4952196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11953
2022-11-23T01:33:43.4952642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4952961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4953388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4953732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4954096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4954552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4954880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4955307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4955649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4956002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4956478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4956805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4957223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4957570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4957901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4958403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4958911Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4959430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4959817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4960172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4960533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkld8ed4
2022-11-23T01:33:43.4960940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkld8ed4/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4961336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kgkt_tb
2022-11-23T01:33:43.4961741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kgkt_tb/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4962103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4962473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkiz0gvg
2022-11-23T01:33:43.4962876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkiz0gvg/_remote_module_non_scriptable.py
2022-11-23T01:33:43.4963253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4963619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4963978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.4964235Z ok (2.375s)
2022-11-23T01:33:43.4964324Z 
2022-11-23T01:33:43.4964527Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4964768Z Ran 1 test in 2.375s
2022-11-23T01:33:43.4964879Z 
2022-11-23T01:33:43.4964939Z OK
2022-11-23T01:33:43.4965029Z 
2022-11-23T01:33:43.4965099Z Generating XML reports...
2022-11-23T01:33:43.4965564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011925.xml
2022-11-23T01:33:43.4966061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4966520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4966867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4967295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4967644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4967890Z 
2022-11-23T01:33:43.4967949Z Running tests...
2022-11-23T01:33:43.4968256Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4968898Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12004
2022-11-23T01:33:43.4969376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12005
2022-11-23T01:33:43.4969711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12006
2022-11-23T01:33:43.4970158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4970486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4970914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4971255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4971587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4972055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4972390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4972803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4973151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4973486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4973941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4974278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4974722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4975235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4975573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4976069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4976587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4977104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4977492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4977828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4978165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4978444Z skip: CUDA is not available. (2.175s)
2022-11-23T01:33:43.4978576Z 
2022-11-23T01:33:43.4978765Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4979003Z Ran 1 test in 2.176s
2022-11-23T01:33:43.4979120Z 
2022-11-23T01:33:43.4979190Z OK (skipped=1)
2022-11-23T01:33:43.4979296Z 
2022-11-23T01:33:43.4979378Z Generating XML reports...
2022-11-23T01:33:43.4979891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011929.xml
2022-11-23T01:33:43.4980390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4980849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4981165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4981593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4981936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4982138Z 
2022-11-23T01:33:43.4982210Z Running tests...
2022-11-23T01:33:43.4982497Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4982931Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12057
2022-11-23T01:33:43.4983368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12058
2022-11-23T01:33:43.4983683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12059
2022-11-23T01:33:43.4984132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4984459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4984885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4985219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4985549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4986020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4986349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4986764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4987106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4987437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.4987887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4988211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4988767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4989115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4989433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.4989923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4990448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4990958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.4991336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.4991688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.4992040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.4992319Z skip: CUDA is not available. (2.075s)
2022-11-23T01:33:43.4992450Z 
2022-11-23T01:33:43.4992634Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4992908Z Ran 1 test in 2.075s
2022-11-23T01:33:43.4993021Z 
2022-11-23T01:33:43.4993093Z OK (skipped=1)
2022-11-23T01:33:43.4993201Z 
2022-11-23T01:33:43.4993271Z Generating XML reports...
2022-11-23T01:33:43.4993732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011933.xml
2022-11-23T01:33:43.4994228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.4994689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4995037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4995466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4995810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4995970Z 
2022-11-23T01:33:43.4996045Z Running tests...
2022-11-23T01:33:43.4996332Z ----------------------------------------------------------------------
2022-11-23T01:33:43.4996783Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12110
2022-11-23T01:33:43.4997226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12111
2022-11-23T01:33:43.4997545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12112
2022-11-23T01:33:43.4998003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.4998334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.4998762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.4999097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.4999433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.4999902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5000229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5000647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5000991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5001325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5001807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5002309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5002637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5003061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5003391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5003751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5004264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5004790Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5005175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5005540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5005874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5006147Z skip: CUDA is not available. (2.172s)
2022-11-23T01:33:43.5006276Z 
2022-11-23T01:33:43.5006461Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5006696Z Ran 1 test in 2.173s
2022-11-23T01:33:43.5006807Z 
2022-11-23T01:33:43.5006877Z OK (skipped=1)
2022-11-23T01:33:43.5006982Z 
2022-11-23T01:33:43.5007065Z Generating XML reports...
2022-11-23T01:33:43.5007509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011937.xml
2022-11-23T01:33:43.5008121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5008580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5008897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5009320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5009659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5009822Z 
2022-11-23T01:33:43.5009895Z Running tests...
2022-11-23T01:33:43.5010180Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5010639Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12163
2022-11-23T01:33:43.5011097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12164
2022-11-23T01:33:43.5011423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12165
2022-11-23T01:33:43.5011863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5012191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5012612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5012946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5013279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5013745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5014069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5014482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5014820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5015377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5015871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5016208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5016618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5016962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5017309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5017834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5018393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5018977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5019365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5019712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5020068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5020343Z skip: CUDA is not available. (2.176s)
2022-11-23T01:33:43.5020476Z 
2022-11-23T01:33:43.5020713Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5020939Z Ran 1 test in 2.177s
2022-11-23T01:33:43.5021047Z 
2022-11-23T01:33:43.5021117Z OK (skipped=1)
2022-11-23T01:33:43.5021222Z 
2022-11-23T01:33:43.5021311Z Generating XML reports...
2022-11-23T01:33:43.5021754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011941.xml
2022-11-23T01:33:43.5022253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5022715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5023106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5023522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5023860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5024028Z 
2022-11-23T01:33:43.5024099Z Running tests...
2022-11-23T01:33:43.5024395Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5024968Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12216
2022-11-23T01:33:43.5025425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12217
2022-11-23T01:33:43.5025788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12218
2022-11-23T01:33:43.5026229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5026557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5026984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5027334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5027657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5028124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5028453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5028875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5029199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5029533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5029996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5030311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5030732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5031072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5031444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5031922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5032434Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5032940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5033320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5033692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5034034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5034310Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.5034442Z 
2022-11-23T01:33:43.5034632Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5034874Z Ran 1 test in 2.174s
2022-11-23T01:33:43.5034985Z 
2022-11-23T01:33:43.5035058Z OK (skipped=1)
2022-11-23T01:33:43.5035164Z 
2022-11-23T01:33:43.5035247Z Generating XML reports...
2022-11-23T01:33:43.5035692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011945.xml
2022-11-23T01:33:43.5036189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5036649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5036966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5037391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5037734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5037896Z 
2022-11-23T01:33:43.5037969Z Running tests...
2022-11-23T01:33:43.5038254Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5038692Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12269
2022-11-23T01:33:43.5039131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12270
2022-11-23T01:33:43.5039458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12271
2022-11-23T01:33:43.5039898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5040232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5040766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5041099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5041430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5041901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5042230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5042638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5042979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5043313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5043766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5044091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5044556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5044899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5045218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5045709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5046225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5046770Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5047139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5047486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5047900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5048165Z skip: CUDA is not available. (2.072s)
2022-11-23T01:33:43.5048297Z 
2022-11-23T01:33:43.5048496Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5048734Z Ran 1 test in 2.073s
2022-11-23T01:33:43.5048847Z 
2022-11-23T01:33:43.5048918Z OK (skipped=1)
2022-11-23T01:33:43.5049011Z 
2022-11-23T01:33:43.5049095Z Generating XML reports...
2022-11-23T01:33:43.5049553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011949.xml
2022-11-23T01:33:43.5050046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5050506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5050821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5051248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5051592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5051755Z 
2022-11-23T01:33:43.5051813Z Running tests...
2022-11-23T01:33:43.5052111Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5052563Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12322
2022-11-23T01:33:43.5053020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12323
2022-11-23T01:33:43.5053335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12324
2022-11-23T01:33:43.5053780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5054108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5054522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5054993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5055450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5055928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5056247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5056697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5057040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5057437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5057896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5058220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5058644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5058972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5059347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5059843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5060357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5060858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5061244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5061587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5061932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5062197Z skip: CUDA is not available. (2.173s)
2022-11-23T01:33:43.5062329Z 
2022-11-23T01:33:43.5062527Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5062766Z Ran 1 test in 2.173s
2022-11-23T01:33:43.5062877Z 
2022-11-23T01:33:43.5062934Z OK (skipped=1)
2022-11-23T01:33:43.5063040Z 
2022-11-23T01:33:43.5063123Z Generating XML reports...
2022-11-23T01:33:43.5063581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011953.xml
2022-11-23T01:33:43.5064073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5064523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5064850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5065277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5065608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5065774Z 
2022-11-23T01:33:43.5065844Z Running tests...
2022-11-23T01:33:43.5066140Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5067040Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:33:43.5067526Z 
2022-11-23T01:33:43.5067722Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5067946Z Ran 1 test in 0.253s
2022-11-23T01:33:43.5068057Z 
2022-11-23T01:33:43.5068126Z OK (skipped=1)
2022-11-23T01:33:43.5068232Z 
2022-11-23T01:33:43.5068316Z Generating XML reports...
2022-11-23T01:33:43.5068760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011956.xml
2022-11-23T01:33:43.5069256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5069719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5070118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5070541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5070883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5071046Z 
2022-11-23T01:33:43.5071116Z Running tests...
2022-11-23T01:33:43.5071402Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5071827Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12385
2022-11-23T01:33:43.5072285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12386
2022-11-23T01:33:43.5072617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12387
2022-11-23T01:33:43.5073060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5073384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5073813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5074156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5074477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5074945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5075270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5075687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5076030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5076365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5076829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5077138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5077562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5077907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5078230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5078938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5079463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5079979Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5080347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5080697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5081036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5081287Z ok (2.275s)
2022-11-23T01:33:43.5081375Z 
2022-11-23T01:33:43.5081576Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5081816Z Ran 1 test in 2.276s
2022-11-23T01:33:43.5081926Z 
2022-11-23T01:33:43.5081986Z OK
2022-11-23T01:33:43.5082075Z 
2022-11-23T01:33:43.5082146Z Generating XML reports...
2022-11-23T01:33:43.5082603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011958.xml
2022-11-23T01:33:43.5083168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5083821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5084297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5084991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5085572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5085932Z 
2022-11-23T01:33:43.5086031Z Running tests...
2022-11-23T01:33:43.5086565Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5087627Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:33:43.5088217Z 
2022-11-23T01:33:43.5088414Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5088653Z Ran 1 test in 0.255s
2022-11-23T01:33:43.5088752Z 
2022-11-23T01:33:43.5088826Z OK (skipped=1)
2022-11-23T01:33:43.5088933Z 
2022-11-23T01:33:43.5089016Z Generating XML reports...
2022-11-23T01:33:43.5089476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012002.xml
2022-11-23T01:33:43.5089961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5090847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5091363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5091808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5092142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5092306Z 
2022-11-23T01:33:43.5092378Z Running tests...
2022-11-23T01:33:43.5092680Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5093084Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12448
2022-11-23T01:33:43.5093499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12449
2022-11-23T01:33:43.5093831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12450
2022-11-23T01:33:43.5094282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5094602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5095189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5095539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5095874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5096333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5096662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5097091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5097435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5097858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5098316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5098640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5099066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5099404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5099722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5100269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5101041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5101917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5102406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5102757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5103101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5103376Z skip: Need at least 3 CUDA devices (2.171s)
2022-11-23T01:33:43.5103514Z 
2022-11-23T01:33:43.5103721Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5103962Z Ran 1 test in 2.172s
2022-11-23T01:33:43.5104074Z 
2022-11-23T01:33:43.5104133Z OK (skipped=1)
2022-11-23T01:33:43.5104239Z 
2022-11-23T01:33:43.5104324Z Generating XML reports...
2022-11-23T01:33:43.5104787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012004.xml
2022-11-23T01:33:43.5105283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5105736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5106068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5106495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5106834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5106988Z 
2022-11-23T01:33:43.5107059Z Running tests...
2022-11-23T01:33:43.5107356Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5107704Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:33:43.5107910Z 
2022-11-23T01:33:43.5108094Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5108328Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5108440Z 
2022-11-23T01:33:43.5108509Z OK (skipped=1)
2022-11-23T01:33:43.5108613Z 
2022-11-23T01:33:43.5108694Z Generating XML reports...
2022-11-23T01:33:43.5109134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012008.xml
2022-11-23T01:33:43.5109624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5110088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5110706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5111147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5111490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5111655Z 
2022-11-23T01:33:43.5111792Z Running tests...
2022-11-23T01:33:43.5112082Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5112392Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.5112790Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12511
2022-11-23T01:33:43.5113168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12512
2022-11-23T01:33:43.5113498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12513
2022-11-23T01:33:43.5113977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5114304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5114715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5115061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5115396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5115865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5116176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5116863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5117435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5117994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5118789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5119126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5119557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5119886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5120218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5120711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5121235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5121752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5122134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5122481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5122814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5123184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcowv09x
2022-11-23T01:33:43.5123572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes6qhxmm
2022-11-23T01:33:43.5123952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_1bs0g8
2022-11-23T01:33:43.5124333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes6qhxmm/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5124734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcowv09x/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5125131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_1bs0g8/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5125392Z ok (2.075s)
2022-11-23T01:33:43.5125563Z 
2022-11-23T01:33:43.5125766Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5126007Z Ran 1 test in 2.075s
2022-11-23T01:33:43.5126119Z 
2022-11-23T01:33:43.5126178Z OK
2022-11-23T01:33:43.5126255Z 
2022-11-23T01:33:43.5126339Z Generating XML reports...
2022-11-23T01:33:43.5126793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012010.xml
2022-11-23T01:33:43.5127284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5127909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5128241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5128673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5129019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5129182Z 
2022-11-23T01:33:43.5129241Z Running tests...
2022-11-23T01:33:43.5129742Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5130280Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.5131009Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12564
2022-11-23T01:33:43.5131689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12565
2022-11-23T01:33:43.5132276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12566
2022-11-23T01:33:43.5133084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5133665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5134387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5135119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5135685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5136524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5137074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5137866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5138492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5139114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5140006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5140619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5141407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5142050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5142657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5143583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5144540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5145521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5146357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5147011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5147696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrg4yffu
2022-11-23T01:33:43.5148452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrg4yffu/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5149173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c3f7n79
2022-11-23T01:33:43.5149895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c3f7n79/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5150558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5151203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraze4gts
2022-11-23T01:33:43.5151876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraze4gts/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5152496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5153141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5153766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5154182Z ok (2.275s)
2022-11-23T01:33:43.5154362Z 
2022-11-23T01:33:43.5154753Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5155173Z Ran 1 test in 2.276s
2022-11-23T01:33:43.5155373Z 
2022-11-23T01:33:43.5155477Z OK
2022-11-23T01:33:43.5155620Z 
2022-11-23T01:33:43.5155767Z Generating XML reports...
2022-11-23T01:33:43.5156598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012014.xml
2022-11-23T01:33:43.5157487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5158283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5158860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5159623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5160251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5160548Z 
2022-11-23T01:33:43.5160664Z Running tests...
2022-11-23T01:33:43.5161204Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5161828Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.5162611Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12617
2022-11-23T01:33:43.5163336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12618
2022-11-23T01:33:43.5163887Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12619
2022-11-23T01:33:43.5164695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5165291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5166103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5166692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5167265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5168204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5168885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5169668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5170273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5170843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5171745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5172730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5173327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5174079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5174688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5175409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5176295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5177274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5178017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5178666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5179283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5179916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxp1vyy7p
2022-11-23T01:33:43.5180622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuy5mhqo
2022-11-23T01:33:43.5181297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvs2_bpz
2022-11-23T01:33:43.5182003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxp1vyy7p/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5182736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuy5mhqo/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5183461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvs2_bpz/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5184142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5184755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5185395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.5185836Z ok (2.075s)
2022-11-23T01:33:43.5186011Z 
2022-11-23T01:33:43.5186396Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5186813Z Ran 1 test in 2.076s
2022-11-23T01:33:43.5187002Z 
2022-11-23T01:33:43.5187103Z OK
2022-11-23T01:33:43.5187267Z 
2022-11-23T01:33:43.5187414Z Generating XML reports...
2022-11-23T01:33:43.5188233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012018.xml
2022-11-23T01:33:43.5189113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5189941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5190524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5191311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5192046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5192333Z 
2022-11-23T01:33:43.5192455Z Running tests...
2022-11-23T01:33:43.5192967Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5193542Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:33:43.5194270Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12670
2022-11-23T01:33:43.5194979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12671
2022-11-23T01:33:43.5195658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12672
2022-11-23T01:33:43.5196516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5197089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5197848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5198469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5199058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5199925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5200509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5201309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5201961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5202527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5203348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5203928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5204687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5205272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5205865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5206780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5207823Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5208749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5209460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5210070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5210737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3i6ii3l
2022-11-23T01:33:43.5211458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3i6ii3l/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5212179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qqq17dl
2022-11-23T01:33:43.5212864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qqq17dl/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5213518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5214171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p8mdefd
2022-11-23T01:33:43.5215181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p8mdefd/_remote_module_non_scriptable.py
2022-11-23T01:33:43.5215708Z ok (2.270s)
2022-11-23T01:33:43.5215886Z 
2022-11-23T01:33:43.5216242Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5216653Z Ran 1 test in 2.271s
2022-11-23T01:33:43.5216835Z 
2022-11-23T01:33:43.5216937Z OK
2022-11-23T01:33:43.5217095Z 
2022-11-23T01:33:43.5217243Z Generating XML reports...
2022-11-23T01:33:43.5218047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012021.xml
2022-11-23T01:33:43.5218939Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5219909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5220493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5221285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5221911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5222194Z 
2022-11-23T01:33:43.5222321Z Running tests...
2022-11-23T01:33:43.5222840Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5223544Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12723
2022-11-23T01:33:43.5224249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12724
2022-11-23T01:33:43.5224849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12725
2022-11-23T01:33:43.5225674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5226260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5227029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5227632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5228245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5229097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5229667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5230397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5231028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5231621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5232525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5233416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5233991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5234760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5235380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5235970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5236901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5237825Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5238574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5239185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5239759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5240489Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5241206Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5241960Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5242810Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5243877Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5244388Z 
2022-11-23T01:33:43.5244863Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5245678Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5246482Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5247296Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5248165Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5248929Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5249708Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5250805Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5251310Z 
2022-11-23T01:33:43.5251790Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5252918Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5253463Z 
2022-11-23T01:33:43.5253920Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5254685Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5255607Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5256392Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5257134Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5257902Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5258705Z STAGE:2022-11-23 01:20:27 12724:12724 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5259519Z STAGE:2022-11-23 01:20:27 12725:12725 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5260360Z STAGE:2022-11-23 01:20:27 12723:12723 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5260867Z ok (2.174s)
2022-11-23T01:33:43.5261043Z 
2022-11-23T01:33:43.5261375Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5261798Z Ran 1 test in 2.174s
2022-11-23T01:33:43.5261985Z 
2022-11-23T01:33:43.5262216Z OK
2022-11-23T01:33:43.5262372Z 
2022-11-23T01:33:43.5262512Z Generating XML reports...
2022-11-23T01:33:43.5263334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012025.xml
2022-11-23T01:33:43.5264244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5265066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5265655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5266536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5267132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5267428Z 
2022-11-23T01:33:43.5267549Z Running tests...
2022-11-23T01:33:43.5268061Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5268752Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12779
2022-11-23T01:33:43.5269500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12780
2022-11-23T01:33:43.5270091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12781
2022-11-23T01:33:43.5270913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5271530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5272241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5272785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5273375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5274470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5275047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5275887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5276609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5277268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5278344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5279353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5279995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5280833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5281589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5282258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5283268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5284274Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5285066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5285997Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5286832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5287453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5288674Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5289180Z 
2022-11-23T01:33:43.5289744Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5290612Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5291534Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5292407Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5293289Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5294191Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5295264Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5296092Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5296930Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5297709Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5298971Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5299512Z 
2022-11-23T01:33:43.5300039Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5300901Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5301730Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5347357Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5348167Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5348957Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5349752Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5350538Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5351361Z STAGE:2022-11-23 01:20:31 12781:12781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5352167Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5352970Z STAGE:2022-11-23 01:20:31 12779:12779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5353769Z STAGE:2022-11-23 01:20:31 12780:12780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5355095Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5356096Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5357634Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5358643Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5360010Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5361103Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5361533Z ok (2.273s)
2022-11-23T01:33:43.5361706Z 
2022-11-23T01:33:43.5362074Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5362471Z Ran 1 test in 2.273s
2022-11-23T01:33:43.5362668Z 
2022-11-23T01:33:43.5362766Z OK
2022-11-23T01:33:43.5362935Z 
2022-11-23T01:33:43.5363075Z Generating XML reports...
2022-11-23T01:33:43.5363914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012029.xml
2022-11-23T01:33:43.5364836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5365693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5366283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5367057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5367770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5368063Z 
2022-11-23T01:33:43.5368191Z Running tests...
2022-11-23T01:33:43.5368717Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5369464Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12835
2022-11-23T01:33:43.5370146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12836
2022-11-23T01:33:43.5370685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12837
2022-11-23T01:33:43.5371469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5371996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5372726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5373320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5373908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5374782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5375617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5376134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5376481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5376810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5377317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5377805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5378128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5378673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5379008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5379325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5379818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5380332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5380749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5381092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5381426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5381781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5382132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5382617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5383004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5383471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5383980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5384441Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5384870Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5385279Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5385702Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5386124Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5386539Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5386966Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5387402Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5387834Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5388257Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5388829Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5389102Z 
2022-11-23T01:33:43.5389340Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5389761Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5390181Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5390600Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5391079Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5391506Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5391923Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5392333Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5392742Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5393167Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5393622Z STAGE:2022-11-23 01:20:35 12837:12837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5394061Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5394482Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5394911Z STAGE:2022-11-23 01:20:35 12835:12835 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5395333Z STAGE:2022-11-23 01:20:35 12836:12836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5396088Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5396635Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5397367Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5397898Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5398608Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5399127Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5399351Z ok (2.175s)
2022-11-23T01:33:43.5399455Z 
2022-11-23T01:33:43.5399647Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5399865Z Ran 1 test in 2.175s
2022-11-23T01:33:43.5399970Z 
2022-11-23T01:33:43.5400031Z OK
2022-11-23T01:33:43.5400119Z 
2022-11-23T01:33:43.5400205Z Generating XML reports...
2022-11-23T01:33:43.5400647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012033.xml
2022-11-23T01:33:43.5401136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5401592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5401904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5402328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5402660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5402826Z 
2022-11-23T01:33:43.5402890Z Running tests...
2022-11-23T01:33:43.5403177Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5403576Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12900
2022-11-23T01:33:43.5404011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12901
2022-11-23T01:33:43.5404337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12902
2022-11-23T01:33:43.5404766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5405086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5405508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5405869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5406192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5406656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5406973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5407379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5407792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5408124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5408577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5408887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5409310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5409642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5409953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5410433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5410948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5411456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5411821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5412165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5412503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5412844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5413200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5413555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5414028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5414526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5415172Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5415654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5416115Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5416757Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5417039Z 
2022-11-23T01:33:43.5417471Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5417760Z 
2022-11-23T01:33:43.5417995Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5418408Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5418873Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5419295Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5419724Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5420164Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5420572Z STAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5420983Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5421577Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5421860Z 
2022-11-23T01:33:43.5422276Z STAGE:2022-11-23 01:20:39 12901:12901 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:39 12902:12902 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5422566Z 
2022-11-23T01:33:43.5423141Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5423669Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5424386Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5424914Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5425134Z ok (2.277s)
2022-11-23T01:33:43.5425234Z 
2022-11-23T01:33:43.5425418Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5425639Z Ran 1 test in 2.277s
2022-11-23T01:33:43.5425753Z 
2022-11-23T01:33:43.5425806Z OK
2022-11-23T01:33:43.5425894Z 
2022-11-23T01:33:43.5425968Z Generating XML reports...
2022-11-23T01:33:43.5426413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012037.xml
2022-11-23T01:33:43.5426897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5427345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5427657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5428071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5428407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5428571Z 
2022-11-23T01:33:43.5428634Z Running tests...
2022-11-23T01:33:43.5428941Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5429344Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12961
2022-11-23T01:33:43.5429738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12962
2022-11-23T01:33:43.5430050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12963
2022-11-23T01:33:43.5430485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5430841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5431260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5431594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5431922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5432378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5432696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5433108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5433436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5433761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5434211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5434528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5434943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5435275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5435587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5436082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5436592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5437095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5437463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5437805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5438229Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5438654Z STAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5438984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5439408Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5439823Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5440410Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5440698Z 
2022-11-23T01:33:43.5440949Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5441589Z STAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5441885Z 
2022-11-23T01:33:43.5442124Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5442696Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5442962Z 
2022-11-23T01:33:43.5443230Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5443654Z STAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5444067Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5444591Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5445029Z STAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5445464Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5445888Z STAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5446300Z STAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5446706Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5447455Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5447931Z 
2022-11-23T01:33:43.5447935Z 
2022-11-23T01:33:43.5448540Z STAGE:2022-11-23 01:20:43 12963:12963 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:43 12961:12961 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:43 12962:12962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5448932Z 
2022-11-23T01:33:43.5448936Z 
2022-11-23T01:33:43.5449499Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5450039Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5450808Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5451732Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5452504Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5453048Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5453283Z ok (2.278s)
2022-11-23T01:33:43.5453384Z 
2022-11-23T01:33:43.5453571Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5453804Z Ran 1 test in 2.279s
2022-11-23T01:33:43.5453909Z 
2022-11-23T01:33:43.5454007Z OK
2022-11-23T01:33:43.5454093Z 
2022-11-23T01:33:43.5454176Z Generating XML reports...
2022-11-23T01:33:43.5454616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012041.xml
2022-11-23T01:33:43.5455267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5455721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5456040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5456534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5456866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5457024Z 
2022-11-23T01:33:43.5457095Z Running tests...
2022-11-23T01:33:43.5457376Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5457786Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13017
2022-11-23T01:33:43.5458206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13018
2022-11-23T01:33:43.5458538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13019
2022-11-23T01:33:43.5458977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5459309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5459743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5460076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5460412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5460883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5461210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5461625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5461969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5462296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5462754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5463075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5463500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5463844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5464164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5464664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5465185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5465703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5466078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5466428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5466810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5467246Z STAGE:2022-11-23 01:20:47 13019:13019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5467665Z STAGE:2022-11-23 01:20:47 13017:13017 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5468091Z STAGE:2022-11-23 01:20:47 13018:13018 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5468852Z STAGE:2022-11-23 01:20:47 13018:13018 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:47 13019:13019 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:47 13017:13017 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5469266Z 
2022-11-23T01:33:43.5469270Z 
2022-11-23T01:33:43.5469878Z STAGE:2022-11-23 01:20:47 13017:13017 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:47 13019:13019 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:47 13018:13018 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5470276Z 
2022-11-23T01:33:43.5470280Z 
2022-11-23T01:33:43.5470843Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5471373Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5472108Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5472641Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5473374Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2511: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5473904Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5474130Z ok (2.177s)
2022-11-23T01:33:43.5474232Z 
2022-11-23T01:33:43.5474429Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5474672Z Ran 1 test in 2.178s
2022-11-23T01:33:43.5474782Z 
2022-11-23T01:33:43.5474830Z OK
2022-11-23T01:33:43.5474920Z 
2022-11-23T01:33:43.5475005Z Generating XML reports...
2022-11-23T01:33:43.5475468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012045.xml
2022-11-23T01:33:43.5475968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5476417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5476748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5477176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5477509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5477673Z 
2022-11-23T01:33:43.5477744Z Running tests...
2022-11-23T01:33:43.5478046Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5478447Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13073
2022-11-23T01:33:43.5478836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13074
2022-11-23T01:33:43.5479196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13075
2022-11-23T01:33:43.5479650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5479976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5480386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5480729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5481063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5481552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5481877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5482304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5482644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5482964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5483435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5483766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5484176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5484519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5484851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5485347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5485861Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5486503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5486891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5487240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5487666Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5488177Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5488528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5488959Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5489386Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5489821Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5490246Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5490676Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5491134Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5491581Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5492014Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5492459Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5492888Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5493484Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5493770Z 
2022-11-23T01:33:43.5494021Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5494754Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5495161Z 
2022-11-23T01:33:43.5495429Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5495872Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5496298Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5496710Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5497307Z STAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5497601Z 
2022-11-23T01:33:43.5497850Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5498475Z STAGE:2022-11-23 01:20:51 13074:13074 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:51 13073:13073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5498782Z 
2022-11-23T01:33:43.5499041Z STAGE:2022-11-23 01:20:51 13075:13075 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5499288Z ok (2.374s)
2022-11-23T01:33:43.5499390Z 
2022-11-23T01:33:43.5499586Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5499827Z Ran 1 test in 2.374s
2022-11-23T01:33:43.5499939Z 
2022-11-23T01:33:43.5499988Z OK
2022-11-23T01:33:43.5500081Z 
2022-11-23T01:33:43.5500165Z Generating XML reports...
2022-11-23T01:33:43.5500621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012049.xml
2022-11-23T01:33:43.5501119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5501575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5501912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5502337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5502670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5502834Z 
2022-11-23T01:33:43.5502905Z Running tests...
2022-11-23T01:33:43.5503201Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5503557Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-11-23T01:33:43.5503769Z 
2022-11-23T01:33:43.5503945Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5504021Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5504025Z 
2022-11-23T01:33:43.5504097Z OK (skipped=1)
2022-11-23T01:33:43.5504102Z 
2022-11-23T01:33:43.5504185Z Generating XML reports...
2022-11-23T01:33:43.5504592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012053.xml
2022-11-23T01:33:43.5504830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5505105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5505231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5505502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5505636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5505679Z 
2022-11-23T01:33:43.5505754Z Running tests...
2022-11-23T01:33:43.5505943Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5506142Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-11-23T01:33:43.5506147Z 
2022-11-23T01:33:43.5506334Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5506407Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5506411Z 
2022-11-23T01:33:43.5506478Z OK (skipped=1)
2022-11-23T01:33:43.5506482Z 
2022-11-23T01:33:43.5506565Z Generating XML reports...
2022-11-23T01:33:43.5506888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012055.xml
2022-11-23T01:33:43.5507116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5507390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5507513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5507793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5507932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5507937Z 
2022-11-23T01:33:43.5508010Z Running tests...
2022-11-23T01:33:43.5508206Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5508431Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13149
2022-11-23T01:33:43.5508589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13150
2022-11-23T01:33:43.5508745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13151
2022-11-23T01:33:43.5509019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5509141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5509419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5509555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5509734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5510006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5510114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5510391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5510527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5510705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5510973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5511123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5511406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5511537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5511699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5511998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5512293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5512631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5512797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5512963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5513124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5513298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5513471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5513632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5513926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5514222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5514509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5514757Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5514994Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5515229Z STAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5515642Z STAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5515650Z 
2022-11-23T01:33:43.5515898Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5516331Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5516337Z 
2022-11-23T01:33:43.5516592Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5516823Z STAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5517071Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5517307Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5517557Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5517800Z STAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5518055Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5518329Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5518586Z STAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5518837Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5519235Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5519271Z 
2022-11-23T01:33:43.5519499Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5519909Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5519915Z 
2022-11-23T01:33:43.5520164Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5520590Z STAGE:2022-11-23 01:20:59 13149:13149 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:20:59 13151:13151 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5520596Z 
2022-11-23T01:33:43.5520851Z STAGE:2022-11-23 01:20:59 13150:13150 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5520918Z ok (2.274s)
2022-11-23T01:33:43.5520923Z 
2022-11-23T01:33:43.5521122Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5521195Z Ran 1 test in 2.274s
2022-11-23T01:33:43.5521200Z 
2022-11-23T01:33:43.5521260Z OK
2022-11-23T01:33:43.5521264Z 
2022-11-23T01:33:43.5521349Z Generating XML reports...
2022-11-23T01:33:43.5521678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012057.xml
2022-11-23T01:33:43.5521911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5522188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5522311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5522593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5522727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5522734Z 
2022-11-23T01:33:43.5522804Z Running tests...
2022-11-23T01:33:43.5523000Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5523229Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13214
2022-11-23T01:33:43.5523375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13215
2022-11-23T01:33:43.5523536Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13216
2022-11-23T01:33:43.5523809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5523930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5524211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5524343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5524520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5524792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5524900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5525207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5525342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5525519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5525790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5525910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5526219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5526353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5526528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5526815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5527114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5527410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5527576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5527819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5527998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5528171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5528336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5528508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5528793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5529087Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5529377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5529627Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5529867Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5530116Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5530361Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5530617Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5530877Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5531106Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5531341Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5531587Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5531827Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5532080Z STAGE:2022-11-23 01:21:03 13215:13215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5532397Z STAGE:2022-11-23 01:21:03 13216:13216 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5532468Z ok (2.378s)
2022-11-23T01:33:43.5532474Z 
2022-11-23T01:33:43.5532671Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5532743Z Ran 1 test in 2.378s
2022-11-23T01:33:43.5532749Z 
2022-11-23T01:33:43.5532796Z OK
2022-11-23T01:33:43.5532800Z 
2022-11-23T01:33:43.5532883Z Generating XML reports...
2022-11-23T01:33:43.5533228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012101.xml
2022-11-23T01:33:43.5533490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5533770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5533894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5534180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5534316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5534321Z 
2022-11-23T01:33:43.5534396Z Running tests...
2022-11-23T01:33:43.5534578Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5534797Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:33:43.5534805Z 
2022-11-23T01:33:43.5535110Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5535185Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5535191Z 
2022-11-23T01:33:43.5535264Z OK (skipped=1)
2022-11-23T01:33:43.5535268Z 
2022-11-23T01:33:43.5535347Z Generating XML reports...
2022-11-23T01:33:43.5535687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012105.xml
2022-11-23T01:33:43.5535921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5536195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5536304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5536585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5536718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5536726Z 
2022-11-23T01:33:43.5536800Z Running tests...
2022-11-23T01:33:43.5536993Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5537215Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:33:43.5537223Z 
2022-11-23T01:33:43.5537413Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5537487Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5537492Z 
2022-11-23T01:33:43.5537548Z OK (skipped=1)
2022-11-23T01:33:43.5537569Z 
2022-11-23T01:33:43.5537637Z Generating XML reports...
2022-11-23T01:33:43.5537979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012106.xml
2022-11-23T01:33:43.5538209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5538486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5538609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5538887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5539064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5539071Z 
2022-11-23T01:33:43.5539144Z Running tests...
2022-11-23T01:33:43.5539324Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5539535Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:33:43.5539540Z 
2022-11-23T01:33:43.5539728Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5539803Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5539841Z 
2022-11-23T01:33:43.5539914Z OK (skipped=1)
2022-11-23T01:33:43.5539918Z 
2022-11-23T01:33:43.5540000Z Generating XML reports...
2022-11-23T01:33:43.5540343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012108.xml
2022-11-23T01:33:43.5540573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5540847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5540956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5541239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5541372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5541377Z 
2022-11-23T01:33:43.5541446Z Running tests...
2022-11-23T01:33:43.5541638Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5541858Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:33:43.5541863Z 
2022-11-23T01:33:43.5542055Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5542128Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5542133Z 
2022-11-23T01:33:43.5542204Z OK (skipped=1)
2022-11-23T01:33:43.5542209Z 
2022-11-23T01:33:43.5542278Z Generating XML reports...
2022-11-23T01:33:43.5542643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012110.xml
2022-11-23T01:33:43.5542872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5543146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5543266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5543548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5543681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5543686Z 
2022-11-23T01:33:43.5543758Z Running tests...
2022-11-23T01:33:43.5543938Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5544181Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13315
2022-11-23T01:33:43.5544338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13316
2022-11-23T01:33:43.5544492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13317
2022-11-23T01:33:43.5544772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5544897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5545178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5545310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5545515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5545777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5545900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5546183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5546316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5546494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5546798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5546920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5547199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5547332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5547495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5547796Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5548095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5548390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5548556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5548717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5548879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5548944Z ok (2.270s)
2022-11-23T01:33:43.5548950Z 
2022-11-23T01:33:43.5549130Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5549202Z Ran 1 test in 2.271s
2022-11-23T01:33:43.5549207Z 
2022-11-23T01:33:43.5549267Z OK
2022-11-23T01:33:43.5549272Z 
2022-11-23T01:33:43.5549354Z Generating XML reports...
2022-11-23T01:33:43.5549694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012111.xml
2022-11-23T01:33:43.5549925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5550199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5550321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5550605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5550727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5550731Z 
2022-11-23T01:33:43.5550802Z Running tests...
2022-11-23T01:33:43.5550996Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5551238Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13368
2022-11-23T01:33:43.5551391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13369
2022-11-23T01:33:43.5551545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13370
2022-11-23T01:33:43.5551819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5551941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5552263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5552386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5552562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5552832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5552956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5553234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5553396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5553572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5553844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5553953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5554230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5554363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5554540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5554840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5555141Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5555434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5555602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5555766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5555916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5556090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5556264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5556439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5556731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5557029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5557317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5557489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:33:43.5557660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:33:43.5557817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:33:43.5558109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:33:43.5558398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:33:43.5558692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-11-23T01:33:43.5558889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:33:43.5559064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:33:43.5559233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:33:43.5559523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:33:43.5559814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:33:43.5560149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-11-23T01:33:43.5560203Z ok (2.376s)
2022-11-23T01:33:43.5560208Z 
2022-11-23T01:33:43.5560401Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5560478Z Ran 1 test in 2.376s
2022-11-23T01:33:43.5560483Z 
2022-11-23T01:33:43.5560543Z OK
2022-11-23T01:33:43.5560547Z 
2022-11-23T01:33:43.5560630Z Generating XML reports...
2022-11-23T01:33:43.5560973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012115.xml
2022-11-23T01:33:43.5561206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5561480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5561592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5561880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5562015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5562020Z 
2022-11-23T01:33:43.5562091Z Running tests...
2022-11-23T01:33:43.5562287Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5562476Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-11-23T01:33:43.5562481Z 
2022-11-23T01:33:43.5562670Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5562744Z Ran 1 test in 0.002s
2022-11-23T01:33:43.5562748Z 
2022-11-23T01:33:43.5562819Z OK (skipped=1)
2022-11-23T01:33:43.5562823Z 
2022-11-23T01:33:43.5562893Z Generating XML reports...
2022-11-23T01:33:43.5563233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012119.xml
2022-11-23T01:33:43.5563465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5563739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5563862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5564141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5564275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5564280Z 
2022-11-23T01:33:43.5564352Z Running tests...
2022-11-23T01:33:43.5564534Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5564784Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13467
2022-11-23T01:33:43.5564941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13468
2022-11-23T01:33:43.5565096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13469
2022-11-23T01:33:43.5565369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5565521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5565803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5565936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5566112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5566371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5566522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5566803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5566936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5567113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5567383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5567506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5567862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5567999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5568160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5568465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5568763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5569061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5569224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5569389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5569553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5569729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5569907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5570064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5570363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5570663Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5570956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5571368Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5571374Z 
2022-11-23T01:33:43.5571614Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5571865Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5572116Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5572392Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5572654Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5572899Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5573152Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5573390Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5573817Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5573822Z 
2022-11-23T01:33:43.5574067Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5574474Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5574480Z 
2022-11-23T01:33:43.5574736Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5575272Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5575279Z 
2022-11-23T01:33:43.5575521Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5575762Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5576007Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5576245Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5576489Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5576728Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5576984Z STAGE:2022-11-23 01:21:23 13467:13467 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5577238Z STAGE:2022-11-23 01:21:23 13469:13469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5577494Z STAGE:2022-11-23 01:21:23 13468:13468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5578070Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5578225Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5578786Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5578936Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5579498Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5579631Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5579748Z ok (2.373s)
2022-11-23T01:33:43.5579754Z 
2022-11-23T01:33:43.5579951Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5580025Z Ran 1 test in 2.373s
2022-11-23T01:33:43.5580031Z 
2022-11-23T01:33:43.5580091Z OK
2022-11-23T01:33:43.5580096Z 
2022-11-23T01:33:43.5580181Z Generating XML reports...
2022-11-23T01:33:43.5580526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012121.xml
2022-11-23T01:33:43.5580765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5581075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5581185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5581469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5581607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5581613Z 
2022-11-23T01:33:43.5581686Z Running tests...
2022-11-23T01:33:43.5581879Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5582128Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13532
2022-11-23T01:33:43.5582284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13533
2022-11-23T01:33:43.5582441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13534
2022-11-23T01:33:43.5582700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5582820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5583103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5583235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5583412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5583685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5583812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5584089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5584223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5584388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5584659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5584776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5585054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5585187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5585362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5585656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5585962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5586259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5586438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5586604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5586762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5586935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5587108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5587278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5587609Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5587905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5588201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5588435Z STAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5588677Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5588915Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5589161Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5589575Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5589581Z 
2022-11-23T01:33:43.5589839Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5590271Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5590277Z 
2022-11-23T01:33:43.5590519Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5590916Z STAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5590924Z 
2022-11-23T01:33:43.5591171Z STAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5591429Z STAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5591838Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5591844Z 
2022-11-23T01:33:43.5592258Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5592275Z 
2022-11-23T01:33:43.5592502Z STAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5592899Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5592906Z 
2022-11-23T01:33:43.5593338Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5593344Z 
2022-11-23T01:33:43.5593591Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5594017Z STAGE:2022-11-23 01:21:27 13534:13534 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:27 13532:13532 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5594022Z 
2022-11-23T01:33:43.5594271Z STAGE:2022-11-23 01:21:27 13533:13533 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5594855Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5595006Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5595655Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5595804Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5596365Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5596510Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5596575Z ok (2.371s)
2022-11-23T01:33:43.5596580Z 
2022-11-23T01:33:43.5596762Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5596840Z Ran 1 test in 2.371s
2022-11-23T01:33:43.5596845Z 
2022-11-23T01:33:43.5596904Z OK
2022-11-23T01:33:43.5596908Z 
2022-11-23T01:33:43.5596990Z Generating XML reports...
2022-11-23T01:33:43.5597331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012125.xml
2022-11-23T01:33:43.5597561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5597841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5597972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5598256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5598377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5598382Z 
2022-11-23T01:33:43.5598451Z Running tests...
2022-11-23T01:33:43.5598645Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5598900Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13597
2022-11-23T01:33:43.5599056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13598
2022-11-23T01:33:43.5599207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13599
2022-11-23T01:33:43.5599479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5599603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5599884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5600005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5600217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5600487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5600608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5600886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5601018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5601220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5601489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5601599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5601880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5602012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5602186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5602484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5602785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5603081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5603245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5603407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5603558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5603730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5603902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5604076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5604366Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5604660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5604953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5605202Z STAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5605442Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5605666Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5605915Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5606327Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5606335Z 
2022-11-23T01:33:43.5606591Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5606845Z STAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5607121Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5607366Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5607842Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5607849Z 
2022-11-23T01:33:43.5608421Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5608471Z 
2022-11-23T01:33:43.5608475Z 
2022-11-23T01:33:43.5608910Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5609170Z STAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5609174Z 
2022-11-23T01:33:43.5609415Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5609811Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5609819Z 
2022-11-23T01:33:43.5610231Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5610236Z 
2022-11-23T01:33:43.5610484Z STAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5610729Z STAGE:2022-11-23 01:21:31 13597:13597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5610982Z STAGE:2022-11-23 01:21:31 13599:13599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5611237Z STAGE:2022-11-23 01:21:31 13598:13598 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5611806Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5611960Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5612522Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5612668Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5613226Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5613372Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5613444Z ok (2.176s)
2022-11-23T01:33:43.5613448Z 
2022-11-23T01:33:43.5613645Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5613706Z Ran 1 test in 2.176s
2022-11-23T01:33:43.5613711Z 
2022-11-23T01:33:43.5613773Z OK
2022-11-23T01:33:43.5613777Z 
2022-11-23T01:33:43.5613863Z Generating XML reports...
2022-11-23T01:33:43.5614230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012129.xml
2022-11-23T01:33:43.5614463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5614739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5614964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5615249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5615424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5615442Z 
2022-11-23T01:33:43.5615502Z Running tests...
2022-11-23T01:33:43.5615699Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5615955Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13662
2022-11-23T01:33:43.5616113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13663
2022-11-23T01:33:43.5616273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13664
2022-11-23T01:33:43.5616546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5616668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5616951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5617075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5617254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5617526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5617647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5617932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5618067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5618246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5618516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5618639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5618903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5619038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5619214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5619514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5619813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5620106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5620274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5620440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5620602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5620798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5620972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5621144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5621438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5621732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5622054Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5622302Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5622542Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5622780Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5623017Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5623264Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5623504Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5623763Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5624195Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5624201Z 
2022-11-23T01:33:43.5624441Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5624843Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5624848Z 
2022-11-23T01:33:43.5625092Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5625505Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5625512Z 
2022-11-23T01:33:43.5625770Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5626023Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5626264Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5626811Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5626829Z 
2022-11-23T01:33:43.5626834Z 
2022-11-23T01:33:43.5627063Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5627314Z STAGE:2022-11-23 01:21:35 13664:13664 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5627566Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5627808Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5628088Z STAGE:2022-11-23 01:21:35 13663:13663 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5628346Z STAGE:2022-11-23 01:21:35 13662:13662 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5628908Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5629060Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5629638Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5629786Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5630345Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5630475Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5630544Z ok (2.274s)
2022-11-23T01:33:43.5630549Z 
2022-11-23T01:33:43.5630742Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5630819Z Ran 1 test in 2.274s
2022-11-23T01:33:43.5630824Z 
2022-11-23T01:33:43.5630883Z OK
2022-11-23T01:33:43.5630887Z 
2022-11-23T01:33:43.5630969Z Generating XML reports...
2022-11-23T01:33:43.5631314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012133.xml
2022-11-23T01:33:43.5631554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5631820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5631943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5632223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5632362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5632366Z 
2022-11-23T01:33:43.5632443Z Running tests...
2022-11-23T01:33:43.5632635Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5632881Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13727
2022-11-23T01:33:43.5633037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13728
2022-11-23T01:33:43.5633194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13729
2022-11-23T01:33:43.5633455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5633576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5633856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5633992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5634172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5634446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5634570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5634884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5635020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5635183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5635484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5635753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5635937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5636215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5636347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5636522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5636820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5637103Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5637267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5637433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5637597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5637774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5637950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5638124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5638423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5638717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5638952Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5639194Z STAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5639447Z STAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5639693Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5639955Z STAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5640215Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5640618Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5640624Z 
2022-11-23T01:33:43.5641035Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5641043Z 
2022-11-23T01:33:43.5641295Z STAGE:2022-11-23 01:21:39 13729:13729 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5641548Z STAGE:2022-11-23 01:21:39 13728:13728 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5641877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5642441Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5642580Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5643127Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5643305Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5643372Z ok (2.376s)
2022-11-23T01:33:43.5643377Z 
2022-11-23T01:33:43.5643576Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5643649Z Ran 1 test in 2.377s
2022-11-23T01:33:43.5643654Z 
2022-11-23T01:33:43.5643713Z OK
2022-11-23T01:33:43.5643717Z 
2022-11-23T01:33:43.5643800Z Generating XML reports...
2022-11-23T01:33:43.5644143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012137.xml
2022-11-23T01:33:43.5644365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5644640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5644765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5645047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5645182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5645187Z 
2022-11-23T01:33:43.5645258Z Running tests...
2022-11-23T01:33:43.5645450Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5645696Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13788
2022-11-23T01:33:43.5645850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13789
2022-11-23T01:33:43.5645990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13790
2022-11-23T01:33:43.5646269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5646390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5646671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5646805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5646981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5647251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5647371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5647640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5647835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5648016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5648291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5648448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5648729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5648865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5649040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5649347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5649630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5649951Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5650116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5650279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5650440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5650618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5650790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5650958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5651251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5651535Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5651788Z STAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5652040Z STAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5652454Z STAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5652459Z 
2022-11-23T01:33:43.5652889Z STAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5652896Z 
2022-11-23T01:33:43.5653138Z STAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5653375Z STAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5653787Z STAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5653792Z 
2022-11-23T01:33:43.5654226Z STAGE:2022-11-23 01:21:43 13789:13789 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:43 13790:13790 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5654231Z 
2022-11-23T01:33:43.5654530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5655188Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5655345Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5655945Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5656096Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5656150Z ok (2.174s)
2022-11-23T01:33:43.5656155Z 
2022-11-23T01:33:43.5656350Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5656424Z Ran 1 test in 2.175s
2022-11-23T01:33:43.5656479Z 
2022-11-23T01:33:43.5656540Z OK
2022-11-23T01:33:43.5656545Z 
2022-11-23T01:33:43.5656628Z Generating XML reports...
2022-11-23T01:33:43.5656969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012141.xml
2022-11-23T01:33:43.5657203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5657479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5657590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5657873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5658008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5658013Z 
2022-11-23T01:33:43.5658082Z Running tests...
2022-11-23T01:33:43.5658277Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5658533Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13849
2022-11-23T01:33:43.5658691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13850
2022-11-23T01:33:43.5658848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13851
2022-11-23T01:33:43.5659119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5659228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5659511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5659645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5659824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5660099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5660222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5660503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5660638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5660815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5661072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5661192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5661469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5661603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5661776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5662071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5662398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5662690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5662858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5663009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5663170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5663374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5663562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5663809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5664104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5664396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5664642Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5664944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5665180Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5665428Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5665670Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5665932Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5666188Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5666428Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5666666Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5666911Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5667153Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5667398Z STAGE:2022-11-23 01:21:47 13851:13851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5667654Z STAGE:2022-11-23 01:21:47 13850:13850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5668218Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5668369Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5668931Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5669082Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5669148Z ok (2.374s)
2022-11-23T01:33:43.5669154Z 
2022-11-23T01:33:43.5669381Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5669458Z Ran 1 test in 2.375s
2022-11-23T01:33:43.5669464Z 
2022-11-23T01:33:43.5669522Z OK
2022-11-23T01:33:43.5669526Z 
2022-11-23T01:33:43.5669598Z Generating XML reports...
2022-11-23T01:33:43.5669940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012145.xml
2022-11-23T01:33:43.5670168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5670444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5670597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5670875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5671009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5671015Z 
2022-11-23T01:33:43.5671087Z Running tests...
2022-11-23T01:33:43.5671270Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5671516Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13910
2022-11-23T01:33:43.5671671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13911
2022-11-23T01:33:43.5671825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13912
2022-11-23T01:33:43.5672101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5672223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5672506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5672639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5672817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5673078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5673197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5673476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5673609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5673787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5674056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5674178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5674457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5674577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5674750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5675044Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5675340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5675633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5675797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5675959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5676159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5676334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5676480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5676651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5676947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5677267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5677557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5677808Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5678049Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5678297Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5678541Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5678798Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5679043Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5679282Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5679521Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5679765Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5680007Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5680262Z STAGE:2022-11-23 01:21:51 13912:13912 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5680514Z STAGE:2022-11-23 01:21:51 13911:13911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5681068Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5681227Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5681778Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5681931Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5682000Z ok (2.273s)
2022-11-23T01:33:43.5682005Z 
2022-11-23T01:33:43.5682201Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5682275Z Ran 1 test in 2.273s
2022-11-23T01:33:43.5682280Z 
2022-11-23T01:33:43.5682338Z OK
2022-11-23T01:33:43.5682345Z 
2022-11-23T01:33:43.5682416Z Generating XML reports...
2022-11-23T01:33:43.5682757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012149.xml
2022-11-23T01:33:43.5682988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5683292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5683418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5683697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5683830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5683835Z 
2022-11-23T01:33:43.5683906Z Running tests...
2022-11-23T01:33:43.5684099Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5684357Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13971
2022-11-23T01:33:43.5684512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13972
2022-11-23T01:33:43.5684666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13973
2022-11-23T01:33:43.5684939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5685059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5685337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5685470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5685647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5685920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5686028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5686307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5686442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5686618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5686884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5687006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5687283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5687417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5687580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5687954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5688253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5688542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5688706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5688869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5689119Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5689360Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5689523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5689749Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5690022Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5690447Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5690453Z 
2022-11-23T01:33:43.5690702Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5690959Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5691246Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5691487Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5691721Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5691955Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5692202Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5692446Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5692689Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5692929Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5693183Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5693437Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5693675Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5694076Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5694081Z 
2022-11-23T01:33:43.5694326Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5694581Z STAGE:2022-11-23 01:21:55 13972:13972 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5695100Z STAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5695109Z 
2022-11-23T01:33:43.5695538Z STAGE:2022-11-23 01:21:55 13973:13973 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:21:55 13971:13971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5695543Z 
2022-11-23T01:33:43.5696113Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5696252Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5696813Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5696960Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5697571Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5697720Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5697787Z ok (2.271s)
2022-11-23T01:33:43.5697793Z 
2022-11-23T01:33:43.5697990Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5698065Z Ran 1 test in 2.271s
2022-11-23T01:33:43.5698070Z 
2022-11-23T01:33:43.5698130Z OK
2022-11-23T01:33:43.5698174Z 
2022-11-23T01:33:43.5698258Z Generating XML reports...
2022-11-23T01:33:43.5698586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012153.xml
2022-11-23T01:33:43.5698819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5699090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5699213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5699493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5699627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5699632Z 
2022-11-23T01:33:43.5699702Z Running tests...
2022-11-23T01:33:43.5699896Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5700164Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14027
2022-11-23T01:33:43.5700308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14028
2022-11-23T01:33:43.5700462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14029
2022-11-23T01:33:43.5700737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5700859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5701138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5701271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5701448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5701719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5701827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5702104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5702238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5702415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5702685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5702805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5703083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5703215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5703391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5703676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5703998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5704287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5704450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5704613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5704776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5705378Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5705526Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5706076Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5706223Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5706779Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5706912Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5706979Z ok (2.173s)
2022-11-23T01:33:43.5706984Z 
2022-11-23T01:33:43.5707180Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5707257Z Ran 1 test in 2.174s
2022-11-23T01:33:43.5707262Z 
2022-11-23T01:33:43.5707324Z OK
2022-11-23T01:33:43.5707329Z 
2022-11-23T01:33:43.5707411Z Generating XML reports...
2022-11-23T01:33:43.5707752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012157.xml
2022-11-23T01:33:43.5707982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5708255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5708366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5708645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5708778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5708783Z 
2022-11-23T01:33:43.5708853Z Running tests...
2022-11-23T01:33:43.5709049Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5709289Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14080
2022-11-23T01:33:43.5709443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14081
2022-11-23T01:33:43.5709596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14082
2022-11-23T01:33:43.5709856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5709978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5710256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5710387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5710592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5710862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5710982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5711263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5711395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5711558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5711854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5711975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5712256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5712387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5712562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5712858Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5713154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5713449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5713601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5713764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5714016Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5714254Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5714412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5714648Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5714893Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5715151Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5715564Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5715569Z 
2022-11-23T01:33:43.5715815Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5716069Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5716311Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5716710Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5716717Z 
2022-11-23T01:33:43.5716963Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5717204Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5717440Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5717720Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5717972Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5718225Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5718452Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5718686Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5718948Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5719193Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5719437Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5719676Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5719929Z STAGE:2022-11-23 01:22:03 14082:14082 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5720180Z STAGE:2022-11-23 01:22:03 14081:14081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5720429Z STAGE:2022-11-23 01:22:03 14080:14080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5720994Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5721134Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5721696Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5721844Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5722395Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5722545Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5722611Z ok (2.273s)
2022-11-23T01:33:43.5722616Z 
2022-11-23T01:33:43.5722813Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5722885Z Ran 1 test in 2.273s
2022-11-23T01:33:43.5722892Z 
2022-11-23T01:33:43.5722951Z OK
2022-11-23T01:33:43.5722956Z 
2022-11-23T01:33:43.5723037Z Generating XML reports...
2022-11-23T01:33:43.5723369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012201.xml
2022-11-23T01:33:43.5723599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5723871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5723992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5724273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5724407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5724412Z 
2022-11-23T01:33:43.5724482Z Running tests...
2022-11-23T01:33:43.5724706Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5724940Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14136
2022-11-23T01:33:43.5725094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14137
2022-11-23T01:33:43.5725246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14138
2022-11-23T01:33:43.5725521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5725669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5725950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5726082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5726262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5726533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5726643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5726924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5727057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5727232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5727507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5727628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5727983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5728115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5728287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5728574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5728874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5729168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5729334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5729495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5729744Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5729985Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5730145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5730413Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5730647Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5730903Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5731146Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5731386Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5731679Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5731933Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5732172Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5732574Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5732611Z 
2022-11-23T01:33:43.5732860Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5733087Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5733340Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5733582Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5733835Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5734087Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5734325Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5734557Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5734793Z STAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5735299Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5735305Z 
2022-11-23T01:33:43.5735551Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5735971Z STAGE:2022-11-23 01:22:07 14138:14138 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:07 14136:14136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5735989Z 
2022-11-23T01:33:43.5736229Z STAGE:2022-11-23 01:22:07 14137:14137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5736792Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5736943Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5737495Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5737643Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5738197Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5738346Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5738413Z ok (2.275s)
2022-11-23T01:33:43.5738418Z 
2022-11-23T01:33:43.5738610Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5738683Z Ran 1 test in 2.276s
2022-11-23T01:33:43.5738734Z 
2022-11-23T01:33:43.5738796Z OK
2022-11-23T01:33:43.5738801Z 
2022-11-23T01:33:43.5738872Z Generating XML reports...
2022-11-23T01:33:43.5739216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012205.xml
2022-11-23T01:33:43.5739448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5739724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5739845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5740160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5740293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5740298Z 
2022-11-23T01:33:43.5740369Z Running tests...
2022-11-23T01:33:43.5740550Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5740792Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14192
2022-11-23T01:33:43.5740947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14193
2022-11-23T01:33:43.5741101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14194
2022-11-23T01:33:43.5741377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5741501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5741781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5741914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5742093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5742354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5742475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5742756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5742889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5743064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5743362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5743632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5743755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5744034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5744154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5744325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5744619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5744916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5745083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5745328Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5745548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5745710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5745944Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5746181Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5746594Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5746626Z 
2022-11-23T01:33:43.5746873Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5747130Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5747388Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5747639Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5747878Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5748113Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5748350Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5748583Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5748824Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5749076Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5749328Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5749571Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5749808Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5750058Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5750295Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5750529Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5750760Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5751000Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5751239Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5751492Z STAGE:2022-11-23 01:22:11 14194:14194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5751744Z STAGE:2022-11-23 01:22:11 14193:14193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5751992Z STAGE:2022-11-23 01:22:11 14192:14192 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5752554Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5752704Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5753291Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5753440Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5753997Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1639: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:33:43.5754170Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-11-23T01:33:43.5754237Z ok (2.373s)
2022-11-23T01:33:43.5754242Z 
2022-11-23T01:33:43.5754437Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5754510Z Ran 1 test in 2.373s
2022-11-23T01:33:43.5754515Z 
2022-11-23T01:33:43.5754576Z OK
2022-11-23T01:33:43.5754581Z 
2022-11-23T01:33:43.5754663Z Generating XML reports...
2022-11-23T01:33:43.5755001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012209.xml
2022-11-23T01:33:43.5755233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5755506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5755618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5755905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5756039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5756044Z 
2022-11-23T01:33:43.5756117Z Running tests...
2022-11-23T01:33:43.5756311Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5756563Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14248
2022-11-23T01:33:43.5756718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14249
2022-11-23T01:33:43.5756869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14250
2022-11-23T01:33:43.5757132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5757256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5757536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5757666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5757842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5758114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5758231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5758512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5758643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5758807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5759080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5759200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5759478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5759640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5759810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5760108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5760403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5760698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5760877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5761038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5761201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5761271Z ok (2.177s)
2022-11-23T01:33:43.5761276Z 
2022-11-23T01:33:43.5761473Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5761546Z Ran 1 test in 2.177s
2022-11-23T01:33:43.5761551Z 
2022-11-23T01:33:43.5761609Z OK
2022-11-23T01:33:43.5761614Z 
2022-11-23T01:33:43.5761696Z Generating XML reports...
2022-11-23T01:33:43.5762023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012213.xml
2022-11-23T01:33:43.5762254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5762532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5762655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5762939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5763072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5763078Z 
2022-11-23T01:33:43.5763147Z Running tests...
2022-11-23T01:33:43.5763339Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5763579Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14301
2022-11-23T01:33:43.5763722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14302
2022-11-23T01:33:43.5763875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14303
2022-11-23T01:33:43.5764145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5764320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5764647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5764781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5764956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5765224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5765346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5765613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5765747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5765920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5766250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5766520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5766641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5766919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5767048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5767209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5767530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5767897Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5768065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5768228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5768386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5768560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5768731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5768902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5769187Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5769482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5769773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5770016Z STAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5770256Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5770488Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5770895Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5770903Z 
2022-11-23T01:33:43.5771147Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5771750Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5771756Z 
2022-11-23T01:33:43.5771760Z 
2022-11-23T01:33:43.5771999Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5772395Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5772402Z 
2022-11-23T01:33:43.5772648Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5772903Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5773168Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5773412Z STAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5773665Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5773918Z STAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5774158Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5774554Z STAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5774587Z 
2022-11-23T01:33:43.5774829Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5775174Z STAGE:2022-11-23 01:22:19 14302:14302 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5775588Z STAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5775594Z 
2022-11-23T01:33:43.5776024Z STAGE:2022-11-23 01:22:19 14301:14301 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:19 14303:14303 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5776031Z 
2022-11-23T01:33:43.5776097Z ok (2.174s)
2022-11-23T01:33:43.5776102Z 
2022-11-23T01:33:43.5776286Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5776360Z Ran 1 test in 2.174s
2022-11-23T01:33:43.5776364Z 
2022-11-23T01:33:43.5776419Z OK
2022-11-23T01:33:43.5776423Z 
2022-11-23T01:33:43.5776506Z Generating XML reports...
2022-11-23T01:33:43.5776847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012217.xml
2022-11-23T01:33:43.5777081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5777354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5777475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5777756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5777880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5777885Z 
2022-11-23T01:33:43.5777956Z Running tests...
2022-11-23T01:33:43.5778145Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5778387Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14366
2022-11-23T01:33:43.5778540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14367
2022-11-23T01:33:43.5778690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14368
2022-11-23T01:33:43.5778962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5779083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5779351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5779487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5779660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5779931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5780099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5780383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5780513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5780688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5780956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5781109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5781389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5781522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5781697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5781992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5782286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5782576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5782738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5782903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5783053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5783227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5783401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5783573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5783864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5784156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5784438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5784688Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5784927Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5785150Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5785395Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5785634Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5785887Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5786132Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5786389Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5786639Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5786876Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5787299Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5787305Z 
2022-11-23T01:33:43.5787546Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5787787Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5788193Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5788230Z 
2022-11-23T01:33:43.5788661Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5788666Z 
2022-11-23T01:33:43.5788909Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5789305Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5789311Z 
2022-11-23T01:33:43.5789555Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5789808Z STAGE:2022-11-23 01:22:23 14368:14368 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5790055Z STAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5790294Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5790722Z STAGE:2022-11-23 01:22:23 14366:14366 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:23 14367:14367 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5790727Z 
2022-11-23T01:33:43.5790793Z ok (2.276s)
2022-11-23T01:33:43.5790798Z 
2022-11-23T01:33:43.5790992Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5791054Z Ran 1 test in 2.276s
2022-11-23T01:33:43.5791059Z 
2022-11-23T01:33:43.5791118Z OK
2022-11-23T01:33:43.5791122Z 
2022-11-23T01:33:43.5791205Z Generating XML reports...
2022-11-23T01:33:43.5791547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012221.xml
2022-11-23T01:33:43.5791778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5792052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5792173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5792454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5792576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5792591Z 
2022-11-23T01:33:43.5792650Z Running tests...
2022-11-23T01:33:43.5792845Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5793088Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14431
2022-11-23T01:33:43.5793245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14432
2022-11-23T01:33:43.5793398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14433
2022-11-23T01:33:43.5793669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5793818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5794097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5794219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5794393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5794660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5794810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5795092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5795224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5795401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5795670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5795778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5796055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5796189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5796361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5796658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5796954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5797245Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5797409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5797571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5797720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5797892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5798066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5798237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5798529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5798824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5799112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5799520Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5799525Z 
2022-11-23T01:33:43.5799765Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5800013Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5800245Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5800517Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5800772Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5801027Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5801276Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5801673Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5801719Z 
2022-11-23T01:33:43.5801958Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5802204Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5802444Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5802696Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5802939Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5803182Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5803432Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5803672Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5803908Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5804143Z STAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5804391Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5804643Z STAGE:2022-11-23 01:22:27 14432:14432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5805058Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5805063Z 
2022-11-23T01:33:43.5805491Z STAGE:2022-11-23 01:22:27 14431:14431 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:27 14433:14433 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5805499Z 
2022-11-23T01:33:43.5805564Z ok (2.277s)
2022-11-23T01:33:43.5805569Z 
2022-11-23T01:33:43.5805752Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5805826Z Ran 1 test in 2.277s
2022-11-23T01:33:43.5805831Z 
2022-11-23T01:33:43.5805892Z OK
2022-11-23T01:33:43.5805897Z 
2022-11-23T01:33:43.5805981Z Generating XML reports...
2022-11-23T01:33:43.5806321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012225.xml
2022-11-23T01:33:43.5806551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5806828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5806951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5807219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5807352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5807356Z 
2022-11-23T01:33:43.5807425Z Running tests...
2022-11-23T01:33:43.5807645Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5807966Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14496
2022-11-23T01:33:43.5808124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14497
2022-11-23T01:33:43.5808277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14498
2022-11-23T01:33:43.5808554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5808675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5808981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5809114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5809286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5809560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5809681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5809959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5810093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5810268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5810527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5810647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5810922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5811058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5811229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5811525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5811821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5812111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5812278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5812429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5812589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5812761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5812933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5813105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5813396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5813687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5813975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5814223Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5814491Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5814714Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5815078Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5815323Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5815566Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5815876Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5816133Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5816384Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5816781Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5816787Z 
2022-11-23T01:33:43.5817025Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5821515Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5821529Z 
2022-11-23T01:33:43.5821774Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5822198Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5822223Z 
2022-11-23T01:33:43.5822465Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5822705Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5822945Z STAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5823178Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5823584Z STAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5823591Z 
2022-11-23T01:33:43.5824186Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:31 14496:14496 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:31 14498:14498 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5824192Z 
2022-11-23T01:33:43.5824196Z 
2022-11-23T01:33:43.5824449Z STAGE:2022-11-23 01:22:31 14497:14497 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5824515Z ok (2.378s)
2022-11-23T01:33:43.5824520Z 
2022-11-23T01:33:43.5824713Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5824785Z Ran 1 test in 2.378s
2022-11-23T01:33:43.5824790Z 
2022-11-23T01:33:43.5824836Z OK
2022-11-23T01:33:43.5824854Z 
2022-11-23T01:33:43.5824924Z Generating XML reports...
2022-11-23T01:33:43.5825267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012229.xml
2022-11-23T01:33:43.5825498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5825844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5825969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5826252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5826387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5826391Z 
2022-11-23T01:33:43.5826460Z Running tests...
2022-11-23T01:33:43.5826643Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5826910Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14561
2022-11-23T01:33:43.5827063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14562
2022-11-23T01:33:43.5827219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14563
2022-11-23T01:33:43.5827497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5827617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5827899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5828033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5828209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5828473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5828594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5828872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5829004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5829180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5829448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5829570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5829848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5829967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5830143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5830436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5830733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5831028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5831191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5831350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5831511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5831685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5831846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5832014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5832334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5832627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5832873Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5833116Z STAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5833363Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5833637Z STAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5833890Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5834134Z STAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5834530Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5834536Z 
2022-11-23T01:33:43.5834781Z STAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5835021Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5835274Z STAGE:2022-11-23 01:22:35 14563:14563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5835529Z STAGE:2022-11-23 01:22:35 14562:14562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5835828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5835894Z ok (2.377s)
2022-11-23T01:33:43.5835901Z 
2022-11-23T01:33:43.5836094Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5836166Z Ran 1 test in 2.377s
2022-11-23T01:33:43.5836171Z 
2022-11-23T01:33:43.5836217Z OK
2022-11-23T01:33:43.5836222Z 
2022-11-23T01:33:43.5836302Z Generating XML reports...
2022-11-23T01:33:43.5836644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012233.xml
2022-11-23T01:33:43.5836875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5837151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5837274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5837552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5837688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5837694Z 
2022-11-23T01:33:43.5837763Z Running tests...
2022-11-23T01:33:43.5837944Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5838177Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14622
2022-11-23T01:33:43.5838331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14623
2022-11-23T01:33:43.5838483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14624
2022-11-23T01:33:43.5838756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5838875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5839153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5839310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5839474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5839743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5839860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5840142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5840300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5840475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5840741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5840862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5841137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5841256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5841429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5841724Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5842020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5842309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5842473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5842630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5842791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5842962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5843123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5843292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5843583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5843873Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5844116Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5844357Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5844598Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5844839Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5845091Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5845333Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5845573Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5845808Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5846078Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5846323Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5846577Z STAGE:2022-11-23 01:22:39 14624:14624 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5846832Z STAGE:2022-11-23 01:22:39 14623:14623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5847128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5847224Z ok (2.276s)
2022-11-23T01:33:43.5847229Z 
2022-11-23T01:33:43.5847412Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5847484Z Ran 1 test in 2.276s
2022-11-23T01:33:43.5847489Z 
2022-11-23T01:33:43.5847548Z OK
2022-11-23T01:33:43.5847552Z 
2022-11-23T01:33:43.5847634Z Generating XML reports...
2022-11-23T01:33:43.5848063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012237.xml
2022-11-23T01:33:43.5848294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5848566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5848688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5848955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5849090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5849095Z 
2022-11-23T01:33:43.5849166Z Running tests...
2022-11-23T01:33:43.5849359Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5849598Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14683
2022-11-23T01:33:43.5849755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14684
2022-11-23T01:33:43.5849909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14685
2022-11-23T01:33:43.5850179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5850299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5850562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5850694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5850870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5851138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5851260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5851538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5851669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5851844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5852111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5852222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5852497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5852625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5852831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5853129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5853423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5853710Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5853874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5854112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5854262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5854434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5854604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5854774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5855392Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5855688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5855936Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5856227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5856472Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5856710Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5856957Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5857213Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5857468Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5857707Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5857944Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5858190Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5858431Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5858686Z STAGE:2022-11-23 01:22:43 14684:14684 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5858927Z STAGE:2022-11-23 01:22:43 14685:14685 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5858992Z ok (2.372s)
2022-11-23T01:33:43.5858998Z 
2022-11-23T01:33:43.5859192Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5859265Z Ran 1 test in 2.373s
2022-11-23T01:33:43.5859270Z 
2022-11-23T01:33:43.5859327Z OK
2022-11-23T01:33:43.5859332Z 
2022-11-23T01:33:43.5859412Z Generating XML reports...
2022-11-23T01:33:43.5859755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012241.xml
2022-11-23T01:33:43.5859985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5860247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5860419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5860704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5860837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5860842Z 
2022-11-23T01:33:43.5860911Z Running tests...
2022-11-23T01:33:43.5861100Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5861331Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14744
2022-11-23T01:33:43.5861522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14745
2022-11-23T01:33:43.5861671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14746
2022-11-23T01:33:43.5861935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5862059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5862334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5862467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5862642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5862937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5863207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5863324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5863594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5863726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5863896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5864164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5864283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5864561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5864695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5864869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5865164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5865447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5865611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5865769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5865939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.5866108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.5866267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5866434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.5866725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5867038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5867318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.5867566Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5867802Z STAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5868051Z STAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5868322Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5868580Z STAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5868836Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5869076Z STAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5869315Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5869715Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5869793Z 
2022-11-23T01:33:43.5870039Z STAGE:2022-11-23 01:22:47 14746:14746 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5870333Z STAGE:2022-11-23 01:22:47 14745:14745 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5870438Z ok (2.374s)
2022-11-23T01:33:43.5870443Z 
2022-11-23T01:33:43.5870665Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5870738Z Ran 1 test in 2.374s
2022-11-23T01:33:43.5870744Z 
2022-11-23T01:33:43.5870803Z OK
2022-11-23T01:33:43.5870807Z 
2022-11-23T01:33:43.5870889Z Generating XML reports...
2022-11-23T01:33:43.5871231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012246.xml
2022-11-23T01:33:43.5871462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5871724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5871848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5872126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5872261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5872266Z 
2022-11-23T01:33:43.5872338Z Running tests...
2022-11-23T01:33:43.5872533Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5872761Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14805
2022-11-23T01:33:43.5872915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14806
2022-11-23T01:33:43.5873056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14807
2022-11-23T01:33:43.5873328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5873451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5873730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5873860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5874062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5874336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5874451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5874726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5874847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5875049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5875321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5875441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5875720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5875851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5876025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5876319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5876615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5876895Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5877192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5877389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5877581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5877865Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5878138Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5878411Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5878687Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5878919Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5879379Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5879385Z 
2022-11-23T01:33:43.5879734Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5880018Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5880288Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5880556Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5880818Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5881098Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5881380Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5881679Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5881924Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5882230Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5882513Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5882780Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5883045Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5883342Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5883782Z STAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5883788Z 
2022-11-23T01:33:43.5884060Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5884548Z STAGE:2022-11-23 01:22:51 14806:14806 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:51 14805:14805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5884554Z 
2022-11-23T01:33:43.5884858Z STAGE:2022-11-23 01:22:51 14807:14807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5884914Z ok (2.175s)
2022-11-23T01:33:43.5884919Z 
2022-11-23T01:33:43.5885146Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5885250Z Ran 1 test in 2.176s
2022-11-23T01:33:43.5885256Z 
2022-11-23T01:33:43.5885354Z OK
2022-11-23T01:33:43.5885359Z 
2022-11-23T01:33:43.5885472Z Generating XML reports...
2022-11-23T01:33:43.5885844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012250.xml
2022-11-23T01:33:43.5886108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5886416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5886592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5886866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5887033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5887038Z 
2022-11-23T01:33:43.5887147Z Running tests...
2022-11-23T01:33:43.5887370Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5887630Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14861
2022-11-23T01:33:43.5887894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14862
2022-11-23T01:33:43.5888107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14863
2022-11-23T01:33:43.5888411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5888521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5888852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5889026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5889233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5889534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5889722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5890032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5890199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5890413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5890673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5890876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5891184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5891346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5891551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5891876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5892232Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5892562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5892757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5892910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5893205Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5893474Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5893667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5893935Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5894209Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5894503Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5894778Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5895155Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5895404Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5895711Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5895984Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5896249Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5896512Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5896822Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5897264Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5897272Z 
2022-11-23T01:33:43.5897555Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5898056Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5898063Z 
2022-11-23T01:33:43.5898353Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5898578Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5898842Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5899126Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5899434Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5899720Z STAGE:2022-11-23 01:22:55 14863:14863 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5899994Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5900279Z STAGE:2022-11-23 01:22:55 14862:14862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5900562Z STAGE:2022-11-23 01:22:55 14861:14861 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5900679Z ok (2.273s)
2022-11-23T01:33:43.5900685Z 
2022-11-23T01:33:43.5900869Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5900979Z Ran 1 test in 2.273s
2022-11-23T01:33:43.5900984Z 
2022-11-23T01:33:43.5901074Z OK
2022-11-23T01:33:43.5901079Z 
2022-11-23T01:33:43.5901222Z Generating XML reports...
2022-11-23T01:33:43.5901592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012254.xml
2022-11-23T01:33:43.5901852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5902160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5902312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5902648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5902771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5902776Z 
2022-11-23T01:33:43.5902874Z Running tests...
2022-11-23T01:33:43.5903096Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5903363Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14917
2022-11-23T01:33:43.5903549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14918
2022-11-23T01:33:43.5903730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14919
2022-11-23T01:33:43.5904033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5904194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5904465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5904648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5904851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5905183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5905335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5905642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5905840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5906049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5906347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5906457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5906784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5906949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5907181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5907509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5907847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5908169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5908364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5908560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5908711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5908857Z skip: CUDA is not available. (2.172s)
2022-11-23T01:33:43.5908862Z 
2022-11-23T01:33:43.5909112Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5909214Z Ran 1 test in 2.172s
2022-11-23T01:33:43.5909219Z 
2022-11-23T01:33:43.5909325Z OK (skipped=1)
2022-11-23T01:33:43.5909330Z 
2022-11-23T01:33:43.5909442Z Generating XML reports...
2022-11-23T01:33:43.5909813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml
2022-11-23T01:33:43.5910078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5910339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5910493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5910820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5910984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5910989Z 
2022-11-23T01:33:43.5911097Z Running tests...
2022-11-23T01:33:43.5911319Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5911594Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14970
2022-11-23T01:33:43.5911778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14971
2022-11-23T01:33:43.5911961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14972
2022-11-23T01:33:43.5912220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5912372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5912725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5912897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5913102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5913428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5913583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5913890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5914052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5914216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5914522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5914721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5915027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5915187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5915391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5915717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5916042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5916321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5916523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5916744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5916959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5917081Z skip: CUDA is not available. (2.173s)
2022-11-23T01:33:43.5917087Z 
2022-11-23T01:33:43.5917313Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5917417Z Ran 1 test in 2.174s
2022-11-23T01:33:43.5917422Z 
2022-11-23T01:33:43.5917521Z OK (skipped=1)
2022-11-23T01:33:43.5917525Z 
2022-11-23T01:33:43.5917644Z Generating XML reports...
2022-11-23T01:33:43.5917973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012301.xml
2022-11-23T01:33:43.5918233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5918533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5918706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5919018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5919184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5919189Z 
2022-11-23T01:33:43.5919289Z Running tests...
2022-11-23T01:33:43.5919520Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5919782Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15023
2022-11-23T01:33:43.5919926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15024
2022-11-23T01:33:43.5920140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15025
2022-11-23T01:33:43.5920442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5920611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5920921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5921121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5921327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5921631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5921741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5922048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5922240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5922444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5922763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5922922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5923230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5923389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5923592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5923879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5924230Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5924557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5924751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5925057Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5925249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5925436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5925713Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5925982Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5926219Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5926490Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5926762Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5927057Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5927358Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5927639Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5927994Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5928264Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5928569Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5928805Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5929088Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5929404Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5929680Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5929986Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5930268Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5930535Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5930829Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5931103Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5931504Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5931551Z 
2022-11-23T01:33:43.5931783Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5932065Z STAGE:2022-11-23 01:23:07 15024:15024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5932346Z STAGE:2022-11-23 01:23:07 15025:15025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5932646Z STAGE:2022-11-23 01:23:07 15023:15023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5932745Z ok (2.274s)
2022-11-23T01:33:43.5932750Z 
2022-11-23T01:33:43.5932976Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5933127Z Ran 1 test in 2.274s
2022-11-23T01:33:43.5933133Z 
2022-11-23T01:33:43.5933221Z OK
2022-11-23T01:33:43.5933226Z 
2022-11-23T01:33:43.5946422Z Generating XML reports...
2022-11-23T01:33:43.5946891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012305.xml
2022-11-23T01:33:43.5947127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5947410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5947524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5947807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5947946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5947954Z 
2022-11-23T01:33:43.5948025Z Running tests...
2022-11-23T01:33:43.5948221Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5948461Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15079
2022-11-23T01:33:43.5948618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15080
2022-11-23T01:33:43.5948771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15081
2022-11-23T01:33:43.5949043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5949152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5949439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5949569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5949744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5950133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5950255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5950533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5950665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5950829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5951100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5951263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5951542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5951673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5951850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5952146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5952444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5952737Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5952890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5953053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5953216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5953307Z skip: CUDA is not available. (2.176s)
2022-11-23T01:33:43.5953314Z 
2022-11-23T01:33:43.5953510Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5953580Z Ran 1 test in 2.177s
2022-11-23T01:33:43.5953585Z 
2022-11-23T01:33:43.5953655Z OK (skipped=1)
2022-11-23T01:33:43.5953660Z 
2022-11-23T01:33:43.5953740Z Generating XML reports...
2022-11-23T01:33:43.5954086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012309.xml
2022-11-23T01:33:43.5954302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5954577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5954698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5954974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5955106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5955111Z 
2022-11-23T01:33:43.5955183Z Running tests...
2022-11-23T01:33:43.5955375Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5955600Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15132
2022-11-23T01:33:43.5955742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15133
2022-11-23T01:33:43.5955890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15134
2022-11-23T01:33:43.5956163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5956280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5956585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5956718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5956893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5957163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5957281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5957545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5957706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5957879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5958147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5958269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5958544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5958675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5958848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5959146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5959434Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5959725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5959887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5960050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5960297Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5960537Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5960699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5960936Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5961183Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5961427Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5961838Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5961845Z 
2022-11-23T01:33:43.5962100Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5962353Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5962593Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5962828Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5963063Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5963307Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5963751Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5963758Z 
2022-11-23T01:33:43.5964004Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5964246Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5964499Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5964737Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5964997Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5965227Z STAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5965475Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5965886Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5965892Z 
2022-11-23T01:33:43.5966142Z STAGE:2022-11-23 01:23:15 15134:15134 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5966568Z STAGE:2022-11-23 01:23:15 15132:15132 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:15 15133:15133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5966575Z 
2022-11-23T01:33:43.5966640Z ok (2.279s)
2022-11-23T01:33:43.5966646Z 
2022-11-23T01:33:43.5966840Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5966902Z Ran 1 test in 2.279s
2022-11-23T01:33:43.5966907Z 
2022-11-23T01:33:43.5966968Z OK
2022-11-23T01:33:43.5966975Z 
2022-11-23T01:33:43.5967055Z Generating XML reports...
2022-11-23T01:33:43.5967395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012313.xml
2022-11-23T01:33:43.5967625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5968013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5968135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5968415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5968536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5968553Z 
2022-11-23T01:33:43.5968611Z Running tests...
2022-11-23T01:33:43.5968802Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5969033Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15188
2022-11-23T01:33:43.5969187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15189
2022-11-23T01:33:43.5969337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15190
2022-11-23T01:33:43.5969609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5969727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5970006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5970127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5970304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5970611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5970732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5971010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5971141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5971321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5971618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5971728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5972008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5972142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5972318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5972615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5972915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5973204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5973371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5973616Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5973766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5973930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5974173Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5974411Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5974825Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5974832Z 
2022-11-23T01:33:43.5975200Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5975454Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5975711Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5975964Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5976204Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5976596Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5976614Z 
2022-11-23T01:33:43.5976845Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5977096Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5977499Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5977554Z 
2022-11-23T01:33:43.5977983Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5977989Z 
2022-11-23T01:33:43.5978225Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5978461Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5978693Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5978974Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5979228Z STAGE:2022-11-23 01:23:19 15190:15190 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5979634Z STAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5979639Z 
2022-11-23T01:33:43.5980069Z STAGE:2022-11-23 01:23:19 15189:15189 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:19 15188:15188 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5980074Z 
2022-11-23T01:33:43.5980139Z ok (2.276s)
2022-11-23T01:33:43.5980144Z 
2022-11-23T01:33:43.5980325Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5980403Z Ran 1 test in 2.277s
2022-11-23T01:33:43.5980408Z 
2022-11-23T01:33:43.5980465Z OK
2022-11-23T01:33:43.5980469Z 
2022-11-23T01:33:43.5980547Z Generating XML reports...
2022-11-23T01:33:43.5980889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012317.xml
2022-11-23T01:33:43.5981120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5981399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5981520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5981792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5981927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5981932Z 
2022-11-23T01:33:43.5982002Z Running tests...
2022-11-23T01:33:43.5982194Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5982431Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15244
2022-11-23T01:33:43.5982583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15245
2022-11-23T01:33:43.5982736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15246
2022-11-23T01:33:43.5983004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5983121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5983389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5983525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5983703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5983972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5984092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5984410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5984545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5984718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5984986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5985093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5985370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5985530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5985703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5986000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5986293Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5986585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5986748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.5986912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.5987064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.5987311Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5987552Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5987796Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5988206Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5988212Z 
2022-11-23T01:33:43.5988457Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5988890Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5988899Z 
2022-11-23T01:33:43.5989149Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5989550Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5989555Z 
2022-11-23T01:33:43.5989792Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5990033Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5990264Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5990500Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5990757Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5991008Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5991282Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5991523Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5991755Z STAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5991988Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.5992228Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5992620Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.5992665Z 
2022-11-23T01:33:43.5992908Z STAGE:2022-11-23 01:23:23 15246:15246 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5993337Z STAGE:2022-11-23 01:23:23 15245:15245 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:23 15244:15244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.5993342Z 
2022-11-23T01:33:43.5993405Z ok (2.173s)
2022-11-23T01:33:43.5993410Z 
2022-11-23T01:33:43.5993603Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5993674Z Ran 1 test in 2.173s
2022-11-23T01:33:43.5993679Z 
2022-11-23T01:33:43.5993735Z OK
2022-11-23T01:33:43.5993739Z 
2022-11-23T01:33:43.5993818Z Generating XML reports...
2022-11-23T01:33:43.5994159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml
2022-11-23T01:33:43.5994392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.5994656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5994779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5995056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5995186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5995191Z 
2022-11-23T01:33:43.5995261Z Running tests...
2022-11-23T01:33:43.5995453Z ----------------------------------------------------------------------
2022-11-23T01:33:43.5995689Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15300
2022-11-23T01:33:43.5995844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15301
2022-11-23T01:33:43.5995995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15302
2022-11-23T01:33:43.5996254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5996376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5996652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5996784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5996960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.5997229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5997353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5997737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5997860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5998068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.5998337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.5998456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.5998730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.5998862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.5999034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.5999360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5999652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.5999932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6000094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6000250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6000415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6000507Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.6000512Z 
2022-11-23T01:33:43.6000703Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6000779Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6000784Z 
2022-11-23T01:33:43.6000852Z OK (skipped=1)
2022-11-23T01:33:43.6000856Z 
2022-11-23T01:33:43.6000939Z Generating XML reports...
2022-11-23T01:33:43.6001267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012325.xml
2022-11-23T01:33:43.6001499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6001771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6001889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6002168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6002296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6002303Z 
2022-11-23T01:33:43.6002372Z Running tests...
2022-11-23T01:33:43.6002566Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6002792Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15353
2022-11-23T01:33:43.6002949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15354
2022-11-23T01:33:43.6003099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15355
2022-11-23T01:33:43.6003371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6003491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6003769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6003898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6004077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6004346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6004456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6004762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6004895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6005070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6005336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6005455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6005760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6005890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6006065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6006353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6006649Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6006945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6007106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6007266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6007427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6007518Z skip: CUDA is not available. (2.178s)
2022-11-23T01:33:43.6007523Z 
2022-11-23T01:33:43.6007780Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6007855Z Ran 1 test in 2.178s
2022-11-23T01:33:43.6007862Z 
2022-11-23T01:33:43.6007920Z OK (skipped=1)
2022-11-23T01:33:43.6007924Z 
2022-11-23T01:33:43.6008008Z Generating XML reports...
2022-11-23T01:33:43.6008349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012329.xml
2022-11-23T01:33:43.6008582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6008855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6008980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6009259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6009393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6009398Z 
2022-11-23T01:33:43.6009456Z Running tests...
2022-11-23T01:33:43.6009657Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6009899Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15406
2022-11-23T01:33:43.6010051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15407
2022-11-23T01:33:43.6010200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15408
2022-11-23T01:33:43.6010470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6010592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6010869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6011001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6011195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6011467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6011586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6011862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6011994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6012169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6012466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6012586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6012852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6012981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6013153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6013450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6013744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6014038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6014199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6014360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6014519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6014601Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.6014619Z 
2022-11-23T01:33:43.6014800Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6014966Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6014972Z 
2022-11-23T01:33:43.6015040Z OK (skipped=1)
2022-11-23T01:33:43.6015045Z 
2022-11-23T01:33:43.6015126Z Generating XML reports...
2022-11-23T01:33:43.6015466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012333.xml
2022-11-23T01:33:43.6015701Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6015973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6016094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6016363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6016497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6016502Z 
2022-11-23T01:33:43.6016571Z Running tests...
2022-11-23T01:33:43.6016761Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6016939Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:33:43.6016944Z 
2022-11-23T01:33:43.6017137Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6017212Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6017217Z 
2022-11-23T01:33:43.6017284Z OK (skipped=1)
2022-11-23T01:33:43.6017289Z 
2022-11-23T01:33:43.6017370Z Generating XML reports...
2022-11-23T01:33:43.6017695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012337.xml
2022-11-23T01:33:43.6017981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6018257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6018380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6018664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6018794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6018837Z 
2022-11-23T01:33:43.6018908Z Running tests...
2022-11-23T01:33:43.6019102Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6019275Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:33:43.6019292Z 
2022-11-23T01:33:43.6019471Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6019542Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6019547Z 
2022-11-23T01:33:43.6019618Z OK (skipped=1)
2022-11-23T01:33:43.6019622Z 
2022-11-23T01:33:43.6019705Z Generating XML reports...
2022-11-23T01:33:43.6020044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012339.xml
2022-11-23T01:33:43.6020274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6020543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6020666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6020934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6021067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6021071Z 
2022-11-23T01:33:43.6021143Z Running tests...
2022-11-23T01:33:43.6021333Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6021516Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-11-23T01:33:43.6021520Z 
2022-11-23T01:33:43.6021705Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6021776Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6021780Z 
2022-11-23T01:33:43.6021848Z OK (skipped=1)
2022-11-23T01:33:43.6021852Z 
2022-11-23T01:33:43.6021933Z Generating XML reports...
2022-11-23T01:33:43.6022261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012340.xml
2022-11-23T01:33:43.6022490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6022760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6022880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6023154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6023284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6023289Z 
2022-11-23T01:33:43.6023358Z Running tests...
2022-11-23T01:33:43.6023544Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6023727Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-11-23T01:33:43.6023745Z 
2022-11-23T01:33:43.6023919Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6023991Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6023995Z 
2022-11-23T01:33:43.6024065Z OK (skipped=1)
2022-11-23T01:33:43.6024070Z 
2022-11-23T01:33:43.6024152Z Generating XML reports...
2022-11-23T01:33:43.6024514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012342.xml
2022-11-23T01:33:43.6024743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6025011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6025129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6025393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6025551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6025555Z 
2022-11-23T01:33:43.6025624Z Running tests...
2022-11-23T01:33:43.6025811Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6025997Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:33:43.6026002Z 
2022-11-23T01:33:43.6026187Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6026258Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6026263Z 
2022-11-23T01:33:43.6026329Z OK (skipped=1)
2022-11-23T01:33:43.6026334Z 
2022-11-23T01:33:43.6026414Z Generating XML reports...
2022-11-23T01:33:43.6026738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012343.xml
2022-11-23T01:33:43.6026965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6027237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6027356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6027635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6027765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6027769Z 
2022-11-23T01:33:43.6027837Z Running tests...
2022-11-23T01:33:43.6028026Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6028219Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-11-23T01:33:43.6028224Z 
2022-11-23T01:33:43.6028403Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6028475Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6028479Z 
2022-11-23T01:33:43.6028544Z OK (skipped=1)
2022-11-23T01:33:43.6028548Z 
2022-11-23T01:33:43.6028629Z Generating XML reports...
2022-11-23T01:33:43.6028968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012345.xml
2022-11-23T01:33:43.6029200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6029470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6029591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6029860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6029990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6029994Z 
2022-11-23T01:33:43.6030066Z Running tests...
2022-11-23T01:33:43.6030257Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6030436Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-11-23T01:33:43.6030442Z 
2022-11-23T01:33:43.6030628Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6030746Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6030752Z 
2022-11-23T01:33:43.6030823Z OK (skipped=1)
2022-11-23T01:33:43.6030827Z 
2022-11-23T01:33:43.6030908Z Generating XML reports...
2022-11-23T01:33:43.6031232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012347.xml
2022-11-23T01:33:43.6031461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6031727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6031874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6032151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6032280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6032285Z 
2022-11-23T01:33:43.6032355Z Running tests...
2022-11-23T01:33:43.6032545Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6032741Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:33:43.6032746Z 
2022-11-23T01:33:43.6032924Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6032995Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6033000Z 
2022-11-23T01:33:43.6033069Z OK (skipped=1)
2022-11-23T01:33:43.6033073Z 
2022-11-23T01:33:43.6033157Z Generating XML reports...
2022-11-23T01:33:43.6033499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012348.xml
2022-11-23T01:33:43.6033730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6034004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6034129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6034406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6034526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6034531Z 
2022-11-23T01:33:43.6034600Z Running tests...
2022-11-23T01:33:43.6034791Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6034995Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6035002Z 
2022-11-23T01:33:43.6035193Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6035266Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6035270Z 
2022-11-23T01:33:43.6035338Z OK (skipped=1)
2022-11-23T01:33:43.6035343Z 
2022-11-23T01:33:43.6035422Z Generating XML reports...
2022-11-23T01:33:43.6035746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012350.xml
2022-11-23T01:33:43.6035978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6036246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6036365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6036639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6036773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6036777Z 
2022-11-23T01:33:43.6036846Z Running tests...
2022-11-23T01:33:43.6037035Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6037275Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6037281Z 
2022-11-23T01:33:43.6037459Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6037530Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6037535Z 
2022-11-23T01:33:43.6037601Z OK (skipped=1)
2022-11-23T01:33:43.6037606Z 
2022-11-23T01:33:43.6037685Z Generating XML reports...
2022-11-23T01:33:43.6038022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012351.xml
2022-11-23T01:33:43.6038251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6038592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6038711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6038992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6039113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6039118Z 
2022-11-23T01:33:43.6039188Z Running tests...
2022-11-23T01:33:43.6039377Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6039590Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:33:43.6039596Z 
2022-11-23T01:33:43.6039783Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6039859Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6039864Z 
2022-11-23T01:33:43.6039933Z OK (skipped=1)
2022-11-23T01:33:43.6039937Z 
2022-11-23T01:33:43.6040018Z Generating XML reports...
2022-11-23T01:33:43.6040353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml
2022-11-23T01:33:43.6040574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6040846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6040967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6041242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6041376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6041381Z 
2022-11-23T01:33:43.6041453Z Running tests...
2022-11-23T01:33:43.6041642Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6041867Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:33:43.6041873Z 
2022-11-23T01:33:43.6042062Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6042123Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6042127Z 
2022-11-23T01:33:43.6042196Z OK (skipped=1)
2022-11-23T01:33:43.6042201Z 
2022-11-23T01:33:43.6042281Z Generating XML reports...
2022-11-23T01:33:43.6042615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012355.xml
2022-11-23T01:33:43.6042841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6043113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6043237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6043515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6043637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6043655Z 
2022-11-23T01:33:43.6043745Z Running tests...
2022-11-23T01:33:43.6043932Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6044146Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6044151Z 
2022-11-23T01:33:43.6044336Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6044406Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6044411Z 
2022-11-23T01:33:43.6044478Z OK (skipped=1)
2022-11-23T01:33:43.6044482Z 
2022-11-23T01:33:43.6044589Z Generating XML reports...
2022-11-23T01:33:43.6044924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012356.xml
2022-11-23T01:33:43.6045140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6045412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6045533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6045810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6045941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6045946Z 
2022-11-23T01:33:43.6046015Z Running tests...
2022-11-23T01:33:43.6046204Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6046427Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:33:43.6046435Z 
2022-11-23T01:33:43.6046622Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6046683Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6046687Z 
2022-11-23T01:33:43.6046757Z OK (skipped=1)
2022-11-23T01:33:43.6046761Z 
2022-11-23T01:33:43.6046843Z Generating XML reports...
2022-11-23T01:33:43.6047182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012358.xml
2022-11-23T01:33:43.6047411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6047775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6047903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6048183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6048319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6048324Z 
2022-11-23T01:33:43.6048382Z Running tests...
2022-11-23T01:33:43.6048576Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6048793Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6048798Z 
2022-11-23T01:33:43.6048985Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6049058Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6049063Z 
2022-11-23T01:33:43.6049132Z OK (skipped=1)
2022-11-23T01:33:43.6049136Z 
2022-11-23T01:33:43.6049217Z Generating XML reports...
2022-11-23T01:33:43.6049553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012359.xml
2022-11-23T01:33:43.6049784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6050042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6050162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6050476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6050611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6050616Z 
2022-11-23T01:33:43.6050686Z Running tests...
2022-11-23T01:33:43.6050877Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6051097Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:33:43.6051102Z 
2022-11-23T01:33:43.6051314Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6051374Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6051389Z 
2022-11-23T01:33:43.6051445Z OK (skipped=1)
2022-11-23T01:33:43.6051450Z 
2022-11-23T01:33:43.6051531Z Generating XML reports...
2022-11-23T01:33:43.6051874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012401.xml
2022-11-23T01:33:43.6052103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6052371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6052491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6052765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6052896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6052904Z 
2022-11-23T01:33:43.6052961Z Running tests...
2022-11-23T01:33:43.6053154Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6053365Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6053370Z 
2022-11-23T01:33:43.6053560Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6053630Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6053635Z 
2022-11-23T01:33:43.6053700Z OK (skipped=1)
2022-11-23T01:33:43.6053704Z 
2022-11-23T01:33:43.6053783Z Generating XML reports...
2022-11-23T01:33:43.6054119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012403.xml
2022-11-23T01:33:43.6054348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6054611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6054730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6055165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6055300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6055305Z 
2022-11-23T01:33:43.6055374Z Running tests...
2022-11-23T01:33:43.6055568Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6055787Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6055793Z 
2022-11-23T01:33:43.6055983Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6056056Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6056062Z 
2022-11-23T01:33:43.6056121Z OK (skipped=1)
2022-11-23T01:33:43.6056125Z 
2022-11-23T01:33:43.6056209Z Generating XML reports...
2022-11-23T01:33:43.6056545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012404.xml
2022-11-23T01:33:43.6056773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6057095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6057217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6057496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6057628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6057633Z 
2022-11-23T01:33:43.6057703Z Running tests...
2022-11-23T01:33:43.6057882Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6058130Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:33:43.6058135Z 
2022-11-23T01:33:43.6058324Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6058397Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6058402Z 
2022-11-23T01:33:43.6058474Z OK (skipped=1)
2022-11-23T01:33:43.6058479Z 
2022-11-23T01:33:43.6058563Z Generating XML reports...
2022-11-23T01:33:43.6058902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012406.xml
2022-11-23T01:33:43.6059128Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6059387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6059508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6059794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6059927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6059931Z 
2022-11-23T01:33:43.6060001Z Running tests...
2022-11-23T01:33:43.6060192Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6060416Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:33:43.6060421Z 
2022-11-23T01:33:43.6060607Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6060679Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6060683Z 
2022-11-23T01:33:43.6060740Z OK (skipped=1)
2022-11-23T01:33:43.6060759Z 
2022-11-23T01:33:43.6060829Z Generating XML reports...
2022-11-23T01:33:43.6061164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012407.xml
2022-11-23T01:33:43.6061393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6061662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6061781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6062059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6062193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6062197Z 
2022-11-23T01:33:43.6062265Z Running tests...
2022-11-23T01:33:43.6062445Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6062663Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6062671Z 
2022-11-23T01:33:43.6062857Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6062928Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6062933Z 
2022-11-23T01:33:43.6062999Z OK (skipped=1)
2022-11-23T01:33:43.6063003Z 
2022-11-23T01:33:43.6063083Z Generating XML reports...
2022-11-23T01:33:43.6063450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012409.xml
2022-11-23T01:33:43.6063681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6063950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6064058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6064337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6064497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6064501Z 
2022-11-23T01:33:43.6064568Z Running tests...
2022-11-23T01:33:43.6064757Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6064985Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-11-23T01:33:43.6064991Z 
2022-11-23T01:33:43.6065179Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6065250Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6065255Z 
2022-11-23T01:33:43.6065323Z OK (skipped=1)
2022-11-23T01:33:43.6065328Z 
2022-11-23T01:33:43.6065396Z Generating XML reports...
2022-11-23T01:33:43.6065730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012411.xml
2022-11-23T01:33:43.6065957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6066229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6066348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6066627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6066758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6066763Z 
2022-11-23T01:33:43.6066831Z Running tests...
2022-11-23T01:33:43.6067011Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6067229Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-11-23T01:33:43.6067234Z 
2022-11-23T01:33:43.6067421Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6067493Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6067498Z 
2022-11-23T01:33:43.6067568Z OK (skipped=1)
2022-11-23T01:33:43.6067572Z 
2022-11-23T01:33:43.6067655Z Generating XML reports...
2022-11-23T01:33:43.6067990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012412.xml
2022-11-23T01:33:43.6068219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6068489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6068597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6068874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6069005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6069010Z 
2022-11-23T01:33:43.6069083Z Running tests...
2022-11-23T01:33:43.6069273Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6069490Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:33:43.6069495Z 
2022-11-23T01:33:43.6069707Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6069778Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6069783Z 
2022-11-23T01:33:43.6069849Z OK (skipped=1)
2022-11-23T01:33:43.6069854Z 
2022-11-23T01:33:43.6069923Z Generating XML reports...
2022-11-23T01:33:43.6070261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012414.xml
2022-11-23T01:33:43.6070488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6070761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6070909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6071188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6071321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6071325Z 
2022-11-23T01:33:43.6071397Z Running tests...
2022-11-23T01:33:43.6071587Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6071809Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15699
2022-11-23T01:33:43.6071963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15700
2022-11-23T01:33:43.6072114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15701
2022-11-23T01:33:43.6072386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6072507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6072783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6072916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6073094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6073355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6073472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6073748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6073878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6074056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6074352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6074621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6074742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6075015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6075134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6075304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6075599Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6075895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6076056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6076258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6076418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6076518Z skip: Need at least 2 CUDA devices (2.172s)
2022-11-23T01:33:43.6076523Z 
2022-11-23T01:33:43.6076715Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6076775Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6076780Z 
2022-11-23T01:33:43.6076848Z OK (skipped=1)
2022-11-23T01:33:43.6076853Z 
2022-11-23T01:33:43.6076933Z Generating XML reports...
2022-11-23T01:33:43.6077273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml
2022-11-23T01:33:43.6077531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6077806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6077930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6078211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6078331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6078348Z 
2022-11-23T01:33:43.6078405Z Running tests...
2022-11-23T01:33:43.6078597Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6078914Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.001s)
2022-11-23T01:33:43.6078921Z 
2022-11-23T01:33:43.6079109Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6079181Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6079186Z 
2022-11-23T01:33:43.6079254Z OK (skipped=1)
2022-11-23T01:33:43.6079258Z 
2022-11-23T01:33:43.6079341Z Generating XML reports...
2022-11-23T01:33:43.6079685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012419.xml
2022-11-23T01:33:43.6079903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6080172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6080293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6080570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6080703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6080708Z 
2022-11-23T01:33:43.6080777Z Running tests...
2022-11-23T01:33:43.6080969Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6081276Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2022-11-23T01:33:43.6081282Z 
2022-11-23T01:33:43.6081472Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6081533Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6081550Z 
2022-11-23T01:33:43.6081606Z OK (skipped=1)
2022-11-23T01:33:43.6081610Z 
2022-11-23T01:33:43.6081692Z Generating XML reports...
2022-11-23T01:33:43.6082030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012421.xml
2022-11-23T01:33:43.6082256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6082529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6082651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6082929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6083090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6083095Z 
2022-11-23T01:33:43.6083153Z Running tests...
2022-11-23T01:33:43.6083343Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6083564Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15772
2022-11-23T01:33:43.6083717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15773
2022-11-23T01:33:43.6083867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15774
2022-11-23T01:33:43.6084169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6084288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6084570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6084690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6084864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6085133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6085250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6085528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6085661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6085835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6086100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6086221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6086486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6086615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6086789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6087089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6087389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6087763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6087933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6088095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6088254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6088308Z ok (3.479s)
2022-11-23T01:33:43.6088314Z 
2022-11-23T01:33:43.6088508Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6088580Z Ran 1 test in 3.480s
2022-11-23T01:33:43.6088585Z 
2022-11-23T01:33:43.6088646Z OK
2022-11-23T01:33:43.6088650Z 
2022-11-23T01:33:43.6088732Z Generating XML reports...
2022-11-23T01:33:43.6089074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012423.xml
2022-11-23T01:33:43.6089304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6089607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6089717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6089996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6090129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6090134Z 
2022-11-23T01:33:43.6090204Z Running tests...
2022-11-23T01:33:43.6090397Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6090622Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15825
2022-11-23T01:33:43.6090805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15826
2022-11-23T01:33:43.6090956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15827
2022-11-23T01:33:43.6091227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6091336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6091613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6091744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6091917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6092185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6092305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6092581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6092713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6092877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6093143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6093266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6093542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6093674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6093848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6094143Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6094442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6094730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6094997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6095146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6095309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6095402Z skip: CUDA is not available. (2.178s)
2022-11-23T01:33:43.6095411Z 
2022-11-23T01:33:43.6095604Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6095678Z Ran 1 test in 2.178s
2022-11-23T01:33:43.6095683Z 
2022-11-23T01:33:43.6095754Z OK (skipped=1)
2022-11-23T01:33:43.6095758Z 
2022-11-23T01:33:43.6095840Z Generating XML reports...
2022-11-23T01:33:43.6096228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012428.xml
2022-11-23T01:33:43.6096447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6096717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6096837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6097113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6097245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6097284Z 
2022-11-23T01:33:43.6097355Z Running tests...
2022-11-23T01:33:43.6097549Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6097780Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15878
2022-11-23T01:33:43.6097936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15879
2022-11-23T01:33:43.6098080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15880
2022-11-23T01:33:43.6098441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6098562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6098839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6098971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6099146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6099415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6099538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6099803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6099934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6100107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6100404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6100674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6100792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6101064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6101196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6101367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6101652Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6101947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6102110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6102273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6102431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6102598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6102803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6102974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6103272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6103552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6103843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6103937Z ok (3.476s)
2022-11-23T01:33:43.6103942Z 
2022-11-23T01:33:43.6104135Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6104206Z Ran 1 test in 3.477s
2022-11-23T01:33:43.6104211Z 
2022-11-23T01:33:43.6104271Z OK
2022-11-23T01:33:43.6104275Z 
2022-11-23T01:33:43.6104353Z Generating XML reports...
2022-11-23T01:33:43.6104695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012432.xml
2022-11-23T01:33:43.6104923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6105184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6105305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6105584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6105721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6105725Z 
2022-11-23T01:33:43.6105793Z Running tests...
2022-11-23T01:33:43.6105984Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6106221Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15940
2022-11-23T01:33:43.6106377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15941
2022-11-23T01:33:43.6106518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15942
2022-11-23T01:33:43.6106785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6106905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6107183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6107319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6107491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6107764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6107882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6108162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6108282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6108459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6108755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6109026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6109146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6109453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6109586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6109758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6110054Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6110335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6110524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6110683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6110844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6110938Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.6110943Z 
2022-11-23T01:33:43.6111139Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6111211Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6111216Z 
2022-11-23T01:33:43.6111285Z OK (skipped=1)
2022-11-23T01:33:43.6111291Z 
2022-11-23T01:33:43.6111361Z Generating XML reports...
2022-11-23T01:33:43.6111699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml
2022-11-23T01:33:43.6111923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6112194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6112317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6112596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6112730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6112735Z 
2022-11-23T01:33:43.6112807Z Running tests...
2022-11-23T01:33:43.6113001Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6113212Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15993
2022-11-23T01:33:43.6113366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15994
2022-11-23T01:33:43.6113518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15995
2022-11-23T01:33:43.6113792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6113912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6114192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6114326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6114500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6114771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6114879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6115157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6115288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6115463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6115730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6115875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6116150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6116283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6116444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6116740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6117033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6117346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6117508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6117671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6117841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6118014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6118174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6118333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6118621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6118911Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6119199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6119266Z ok (3.075s)
2022-11-23T01:33:43.6119272Z 
2022-11-23T01:33:43.6119466Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6119538Z Ran 1 test in 3.075s
2022-11-23T01:33:43.6119543Z 
2022-11-23T01:33:43.6119602Z OK
2022-11-23T01:33:43.6119607Z 
2022-11-23T01:33:43.6119688Z Generating XML reports...
2022-11-23T01:33:43.6120017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012441.xml
2022-11-23T01:33:43.6120246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6120518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6120641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6120921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6121057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6121062Z 
2022-11-23T01:33:43.6121131Z Running tests...
2022-11-23T01:33:43.6121324Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6121555Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16052
2022-11-23T01:33:43.6121697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16053
2022-11-23T01:33:43.6121851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16054
2022-11-23T01:33:43.6122121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6122242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6122562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6122695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6122874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6123142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6123250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6123528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6123689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6123862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6124131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6124253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6124527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6124657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6124831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6125115Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6125411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6125702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6125868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6126030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6126190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6126284Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.6126290Z 
2022-11-23T01:33:43.6126485Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6126559Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6126564Z 
2022-11-23T01:33:43.6126626Z OK (skipped=1)
2022-11-23T01:33:43.6126631Z 
2022-11-23T01:33:43.6126712Z Generating XML reports...
2022-11-23T01:33:43.6127051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml
2022-11-23T01:33:43.6127279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6127552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6127673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6128027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6128161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6128166Z 
2022-11-23T01:33:43.6128237Z Running tests...
2022-11-23T01:33:43.6128418Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6128660Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16105
2022-11-23T01:33:43.6128814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16106
2022-11-23T01:33:43.6128966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16107
2022-11-23T01:33:43.6129272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6129393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6129673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6129805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6129967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6130263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6130384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6130660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6130792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6130969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6131236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6131353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6131630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6131750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6131922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6132216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6132510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6132801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6132963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6133119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6133277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6133449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6133610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6133779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6134069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6134357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6134641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6134707Z ok (3.276s)
2022-11-23T01:33:43.6134712Z 
2022-11-23T01:33:43.6135005Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6135080Z Ran 1 test in 3.276s
2022-11-23T01:33:43.6135085Z 
2022-11-23T01:33:43.6135143Z OK
2022-11-23T01:33:43.6135148Z 
2022-11-23T01:33:43.6135217Z Generating XML reports...
2022-11-23T01:33:43.6135558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml
2022-11-23T01:33:43.6135836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6136109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6136229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6136509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6136642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6136647Z 
2022-11-23T01:33:43.6136752Z Running tests...
2022-11-23T01:33:43.6136934Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6137171Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16167
2022-11-23T01:33:43.6137324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16168
2022-11-23T01:33:43.6137479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16169
2022-11-23T01:33:43.6137749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6137868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6138147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6138279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6138458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6138716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6138836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6139114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6139244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6139419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6139685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6139805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6140080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6140203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6140377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6140674Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6140967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6141252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6141414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6141573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6141734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6141905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6142066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6142419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6142587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6142879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6143168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6143234Z ok (3.377s)
2022-11-23T01:33:43.6143266Z 
2022-11-23T01:33:43.6143465Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6143539Z Ran 1 test in 3.378s
2022-11-23T01:33:43.6143545Z 
2022-11-23T01:33:43.6143600Z OK
2022-11-23T01:33:43.6143604Z 
2022-11-23T01:33:43.6143675Z Generating XML reports...
2022-11-23T01:33:43.6144017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012455.xml
2022-11-23T01:33:43.6144249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6144525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6144645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6144921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6145053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6145060Z 
2022-11-23T01:33:43.6145130Z Running tests...
2022-11-23T01:33:43.6145321Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6145544Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16229
2022-11-23T01:33:43.6145702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16230
2022-11-23T01:33:43.6145855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16231
2022-11-23T01:33:43.6146125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6146243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6146523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6146656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6146828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6147096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6147205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6147482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6147613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6147787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6148054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6148172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6148450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6148580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6148740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6149071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6149370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6149660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6149822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6150010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6150166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6150339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6150509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6150790Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6150954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6151245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6151536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6151604Z ok (7.180s)
2022-11-23T01:33:43.6151609Z 
2022-11-23T01:33:43.6151802Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6151875Z Ran 1 test in 7.180s
2022-11-23T01:33:43.6151880Z 
2022-11-23T01:33:43.6151939Z OK
2022-11-23T01:33:43.6151943Z 
2022-11-23T01:33:43.6152023Z Generating XML reports...
2022-11-23T01:33:43.6152354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012500.xml
2022-11-23T01:33:43.6152584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6152855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6152975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6153255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6153388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6153393Z 
2022-11-23T01:33:43.6153460Z Running tests...
2022-11-23T01:33:43.6153650Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6153889Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16288
2022-11-23T01:33:43.6154032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16289
2022-11-23T01:33:43.6154184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16290
2022-11-23T01:33:43.6154453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6154573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6154850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6154982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6155157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6155455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6155564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6155838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6155971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6156142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6156405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6156555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6156831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6156961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6157135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6157423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6157718Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6158006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6158171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6158335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6158492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6158557Z ok (2.176s)
2022-11-23T01:33:43.6158563Z 
2022-11-23T01:33:43.6158759Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6158820Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6158835Z 
2022-11-23T01:33:43.6158881Z OK
2022-11-23T01:33:43.6158886Z 
2022-11-23T01:33:43.6158967Z Generating XML reports...
2022-11-23T01:33:43.6159304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012509.xml
2022-11-23T01:33:43.6159535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6159810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6159932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6160210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6160342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6160349Z 
2022-11-23T01:33:43.6160408Z Running tests...
2022-11-23T01:33:43.6160603Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6160842Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16341
2022-11-23T01:33:43.6160995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16342
2022-11-23T01:33:43.6161144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16343
2022-11-23T01:33:43.6161416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6161534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6161813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6161969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6162133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6162401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6162520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6162796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6162926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6163128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6163396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6163516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6163781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6163912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6164083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6164378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6164671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6164960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6165123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6165281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6165440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6165494Z ok (2.173s)
2022-11-23T01:33:43.6165509Z 
2022-11-23T01:33:43.6165690Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6165761Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6165765Z 
2022-11-23T01:33:43.6165821Z OK
2022-11-23T01:33:43.6165825Z 
2022-11-23T01:33:43.6165907Z Generating XML reports...
2022-11-23T01:33:43.6166244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012513.xml
2022-11-23T01:33:43.6166477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6166747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6166868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6167133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6167264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6167270Z 
2022-11-23T01:33:43.6167339Z Running tests...
2022-11-23T01:33:43.6167531Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6167808Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6167817Z 
2022-11-23T01:33:43.6168010Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6168084Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6168089Z 
2022-11-23T01:33:43.6168158Z OK (skipped=1)
2022-11-23T01:33:43.6168163Z 
2022-11-23T01:33:43.6168246Z Generating XML reports...
2022-11-23T01:33:43.6168615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012516.xml
2022-11-23T01:33:43.6168847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6169119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6169238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6169517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6169681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6169686Z 
2022-11-23T01:33:43.6169753Z Running tests...
2022-11-23T01:33:43.6169944Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6170116Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6170132Z 
2022-11-23T01:33:43.6170322Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6170392Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6170397Z 
2022-11-23T01:33:43.6170462Z OK (skipped=1)
2022-11-23T01:33:43.6170467Z 
2022-11-23T01:33:43.6170548Z Generating XML reports...
2022-11-23T01:33:43.6170889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012518.xml
2022-11-23T01:33:43.6171119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6171393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6171512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6171778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6171910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6171916Z 
2022-11-23T01:33:43.6171985Z Running tests...
2022-11-23T01:33:43.6172172Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6172370Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6172375Z 
2022-11-23T01:33:43.6172564Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6172634Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6172639Z 
2022-11-23T01:33:43.6172710Z OK (skipped=1)
2022-11-23T01:33:43.6172715Z 
2022-11-23T01:33:43.6172796Z Generating XML reports...
2022-11-23T01:33:43.6173121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012520.xml
2022-11-23T01:33:43.6173350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6173621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6173739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6174016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6174148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6174153Z 
2022-11-23T01:33:43.6174222Z Running tests...
2022-11-23T01:33:43.6174414Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6174590Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6174606Z 
2022-11-23T01:33:43.6174780Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6174946Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6174952Z 
2022-11-23T01:33:43.6175022Z OK (skipped=1)
2022-11-23T01:33:43.6175074Z 
2022-11-23T01:33:43.6175159Z Generating XML reports...
2022-11-23T01:33:43.6175499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012521.xml
2022-11-23T01:33:43.6175731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6176005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6176125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6176430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6176564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6176568Z 
2022-11-23T01:33:43.6176639Z Running tests...
2022-11-23T01:33:43.6176829Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6177019Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6177024Z 
2022-11-23T01:33:43.6177213Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6177288Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6177293Z 
2022-11-23T01:33:43.6177361Z OK (skipped=1)
2022-11-23T01:33:43.6177365Z 
2022-11-23T01:33:43.6177446Z Generating XML reports...
2022-11-23T01:33:43.6177772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012523.xml
2022-11-23T01:33:43.6178004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6178278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6178396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6178675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6178804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6178809Z 
2022-11-23T01:33:43.6178877Z Running tests...
2022-11-23T01:33:43.6179067Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6179252Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6179268Z 
2022-11-23T01:33:43.6179447Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6179520Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6179524Z 
2022-11-23T01:33:43.6179593Z OK (skipped=1)
2022-11-23T01:33:43.6179598Z 
2022-11-23T01:33:43.6179678Z Generating XML reports...
2022-11-23T01:33:43.6180012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml
2022-11-23T01:33:43.6180240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6180511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6180629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6180897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6181028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6181035Z 
2022-11-23T01:33:43.6181103Z Running tests...
2022-11-23T01:33:43.6181292Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6181479Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6181484Z 
2022-11-23T01:33:43.6181701Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6181774Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6181779Z 
2022-11-23T01:33:43.6181848Z OK (skipped=1)
2022-11-23T01:33:43.6181852Z 
2022-11-23T01:33:43.6181930Z Generating XML reports...
2022-11-23T01:33:43.6182252Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012526.xml
2022-11-23T01:33:43.6182483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6182753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6182900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6183179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6183312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6183317Z 
2022-11-23T01:33:43.6183385Z Running tests...
2022-11-23T01:33:43.6183576Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6183765Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:33:43.6183770Z 
2022-11-23T01:33:43.6183945Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6184017Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6184021Z 
2022-11-23T01:33:43.6184089Z OK (skipped=1)
2022-11-23T01:33:43.6184096Z 
2022-11-23T01:33:43.6184177Z Generating XML reports...
2022-11-23T01:33:43.6184518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012528.xml
2022-11-23T01:33:43.6184747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6185020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6185140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6185406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6185537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6185542Z 
2022-11-23T01:33:43.6185609Z Running tests...
2022-11-23T01:33:43.6185800Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6186023Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16474
2022-11-23T01:33:43.6186178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16475
2022-11-23T01:33:43.6186331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16476
2022-11-23T01:33:43.6186602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6186720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6186988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6187119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6187293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6187562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6187684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6187968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6188124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6188301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6188570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6188678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6188954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6189084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6189288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6189584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6189881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6190171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6190333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6190497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6190735Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6190982Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6191141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6191378Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6191624Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6191862Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6192116Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6192359Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6192601Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6192852Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6193092Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6193329Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6193562Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6193802Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6194052Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6194294Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6194538Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6194970Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6194977Z 
2022-11-23T01:33:43.6195205Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6195465Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6195710Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6195945Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6196197Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6196440Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6196707Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6197134Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6197141Z 
2022-11-23T01:33:43.6197381Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6197618Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6197839Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6198081Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6198328Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6198568Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6198984Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6198989Z 
2022-11-23T01:33:43.6199241Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6199480Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6199715Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6199944Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6200185Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6200430Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6200670Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6200921Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6201164Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6201413Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6201648Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6201881Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6202115Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6202523Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6202529Z 
2022-11-23T01:33:43.6202798Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6203218Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6203233Z 
2022-11-23T01:33:43.6203474Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6203712Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6203977Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6204211Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6204452Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6204702Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6205110Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6205116Z 
2022-11-23T01:33:43.6205366Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6205617Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6205847Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6206081Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6206315Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6206556Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6206805Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6207046Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6207463Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6207471Z 
2022-11-23T01:33:43.6207792Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6208035Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6208273Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6208492Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6208738Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6208993Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6209240Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6209480Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6209911Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6209916Z 
2022-11-23T01:33:43.6210186Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6210584Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6210589Z 
2022-11-23T01:33:43.6210831Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6211409Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6211443Z 
2022-11-23T01:33:43.6211447Z 
2022-11-23T01:33:43.6211877Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6211883Z 
2022-11-23T01:33:43.6212121Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6212356Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6212577Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6212821Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6213064Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6213300Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6213553Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6213803Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6214059Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6214297Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6214531Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6214750Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6215086Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6215493Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6215498Z 
2022-11-23T01:33:43.6215749Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6216176Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6216181Z 
2022-11-23T01:33:43.6216418Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6216655Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6216889Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6217137Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6217446Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6217680Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6217921Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6218172Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6218421Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6218696Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6218931Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6219163Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6219408Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6219650Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6220072Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6220078Z 
2022-11-23T01:33:43.6220319Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6220576Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6220813Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6221049Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6221283Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6221526Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6221764Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6222016Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6222257Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6222497Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6222750Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6222988Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6223222Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6223450Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6223692Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6223932Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6224346Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6224354Z 
2022-11-23T01:33:43.6224606Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6224889Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6225116Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6225348Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6225579Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6225821Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6226085Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6226676Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6226682Z 
2022-11-23T01:33:43.6226687Z 
2022-11-23T01:33:43.6226939Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6227177Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6227416Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6227650Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6227898Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6228139Z STAGE:2022-11-23 01:25:31 16475:16475 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6228385Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6228629Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6228882Z STAGE:2022-11-23 01:25:31 16476:16476 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6229130Z STAGE:2022-11-23 01:25:31 16474:16474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6229197Z ok (2.274s)
2022-11-23T01:33:43.6229202Z 
2022-11-23T01:33:43.6229395Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6229467Z Ran 1 test in 2.274s
2022-11-23T01:33:43.6229474Z 
2022-11-23T01:33:43.6229523Z OK
2022-11-23T01:33:43.6229540Z 
2022-11-23T01:33:43.6229611Z Generating XML reports...
2022-11-23T01:33:43.6229954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012529.xml
2022-11-23T01:33:43.6230186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6230460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6230582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6230867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6230999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6231004Z 
2022-11-23T01:33:43.6231072Z Running tests...
2022-11-23T01:33:43.6231256Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6231969Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:33:43.6231975Z 
2022-11-23T01:33:43.6232169Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6232242Z Ran 1 test in 0.255s
2022-11-23T01:33:43.6232247Z 
2022-11-23T01:33:43.6232316Z OK (skipped=1)
2022-11-23T01:33:43.6232321Z 
2022-11-23T01:33:43.6232404Z Generating XML reports...
2022-11-23T01:33:43.6232744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012533.xml
2022-11-23T01:33:43.6232979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6233279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6233398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6233669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6233801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6233806Z 
2022-11-23T01:33:43.6233873Z Running tests...
2022-11-23T01:33:43.6234067Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6234300Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16540
2022-11-23T01:33:43.6234456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16541
2022-11-23T01:33:43.6234611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16542
2022-11-23T01:33:43.6234882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6234992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6235272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6235416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6235592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6235859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6235978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6236243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6236375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6236549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6236847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6237114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6237234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6237515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6237646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6237815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6238100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6238397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6238590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6238750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6238908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6239081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6239249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6239420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6239744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6240026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6240315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6240563Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6240806Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6241043Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6241289Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6241532Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6241953Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6241959Z 
2022-11-23T01:33:43.6242217Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6242469Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6242697Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6242938Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6243172Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6243419Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6243660Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6243915Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6244158Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6244409Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6244658Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6244887Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6245118Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6245358Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6245610Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6245878Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6246121Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6246359Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6246611Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6246858Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6247147Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6247380Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6247609Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6247928Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6248342Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6248596Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6248601Z 
2022-11-23T01:33:43.6248853Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6249104Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6249342Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6249575Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6249797Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6250041Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6250283Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6250520Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6250949Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6250956Z 
2022-11-23T01:33:43.6251209Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6251446Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6251677Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6251909Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6252314Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6252320Z 
2022-11-23T01:33:43.6252550Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6252805Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6253052Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6253329Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6253566Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6253802Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6254032Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6254274Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6254539Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6254764Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6255109Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6255361Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6255609Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6255849Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6256084Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6256320Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6256561Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6256814Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6257047Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6257289Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6257536Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6257785Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6258021Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6258254Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6258488Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6258731Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6259144Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6259150Z 
2022-11-23T01:33:43.6259400Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6259813Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6259829Z 
2022-11-23T01:33:43.6260057Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6260293Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6260526Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6260816Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6261225Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6261230Z 
2022-11-23T01:33:43.6261484Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6261908Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6261948Z 
2022-11-23T01:33:43.6262185Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6262420Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6262659Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6262891Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6263311Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6263316Z 
2022-11-23T01:33:43.6263558Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6263812Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6264066Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6264303Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6264539Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6264770Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6265014Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6265265Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6265495Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6265736Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6266159Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6266165Z 
2022-11-23T01:33:43.6266403Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6266635Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6266868Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6267110Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6267359Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6267599Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6267851Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6268124Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6268377Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6268613Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6268846Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6269078Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6269318Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6269764Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6269769Z 
2022-11-23T01:33:43.6270015Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6270267Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6270519Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6270748Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6270984Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6271223Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6271464Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6271714Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6271954Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6272192Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6272443Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6272694Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6272920Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6273155Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6273387Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6273630Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6273883Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6274119Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6274370Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6274605Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6274855Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6275082Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6275316Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6275545Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6275817Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6276068Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6276476Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6276481Z 
2022-11-23T01:33:43.6276902Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6276932Z 
2022-11-23T01:33:43.6277168Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6277400Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6277633Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6277864Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6278102Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6278353Z STAGE:2022-11-23 01:25:37 16542:16542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6278596Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6278848Z STAGE:2022-11-23 01:25:37 16540:16540 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6279097Z STAGE:2022-11-23 01:25:37 16541:16541 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6279162Z ok (2.176s)
2022-11-23T01:33:43.6279169Z 
2022-11-23T01:33:43.6279365Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6279436Z Ran 1 test in 2.177s
2022-11-23T01:33:43.6279441Z 
2022-11-23T01:33:43.6279488Z OK
2022-11-23T01:33:43.6279492Z 
2022-11-23T01:33:43.6279574Z Generating XML reports...
2022-11-23T01:33:43.6279915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012535.xml
2022-11-23T01:33:43.6280147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6280425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6280546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6280827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6280962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6280967Z 
2022-11-23T01:33:43.6281036Z Running tests...
2022-11-23T01:33:43.6281218Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6281447Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16605
2022-11-23T01:33:43.6281601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16606
2022-11-23T01:33:43.6281754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16607
2022-11-23T01:33:43.6282029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6282149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6282427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6282588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6282754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6283050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6283321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6283443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6283751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6283883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6284054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6284326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6284444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6284716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6284848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6285021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6285317Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6285612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6285772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6285933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6286106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6286277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6286423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6286594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6286886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6287178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6287468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6287776Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6288015Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6288260Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6288501Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6288746Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6289001Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6289241Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6289506Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6289751Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6289990Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6290247Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6290499Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6290764Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6290989Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6291234Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6291477Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6291727Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6291981Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6292219Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6292452Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6292694Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6292933Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6293172Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6293421Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6293657Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6293895Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6294135Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6294375Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6294627Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6294974Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6295217Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6295440Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6295685Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6295935Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6296175Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6296424Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6296665Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6296899Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6297186Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6297427Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6297667Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6297916Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6298151Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6298381Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6298656Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6298905Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6299146Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6299396Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6299635Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6299860Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6300105Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6300345Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6300592Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6300846Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6301085Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6301321Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6301559Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6301977Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6301986Z 
2022-11-23T01:33:43.6302226Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6302463Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6302692Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6302935Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6303171Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6303422Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6303672Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6303908Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6304142Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6304374Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6304637Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6304890Z STAGE:2022-11-23 01:25:41 16607:16607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6305142Z STAGE:2022-11-23 01:25:41 16606:16606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6305210Z ok (2.279s)
2022-11-23T01:33:43.6305215Z 
2022-11-23T01:33:43.6305408Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6305480Z Ran 1 test in 2.280s
2022-11-23T01:33:43.6305485Z 
2022-11-23T01:33:43.6305543Z OK
2022-11-23T01:33:43.6305580Z 
2022-11-23T01:33:43.6305663Z Generating XML reports...
2022-11-23T01:33:43.6305996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012539.xml
2022-11-23T01:33:43.6306227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6306504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6306624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6306903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6307037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6307043Z 
2022-11-23T01:33:43.6307111Z Running tests...
2022-11-23T01:33:43.6307304Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6307526Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16666
2022-11-23T01:33:43.6307678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16667
2022-11-23T01:33:43.6307828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16668
2022-11-23T01:33:43.6308106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6308227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6308506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6308637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6308811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6309084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6309195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6309472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6309606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6309781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6310050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6310168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6310446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6310577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6310752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6311038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6311360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6311658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6311820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6311982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6312142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6312232Z skip: CUDA is not available. (2.175s)
2022-11-23T01:33:43.6312264Z 
2022-11-23T01:33:43.6312456Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6312527Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6312532Z 
2022-11-23T01:33:43.6312589Z OK (skipped=1)
2022-11-23T01:33:43.6312594Z 
2022-11-23T01:33:43.6312677Z Generating XML reports...
2022-11-23T01:33:43.6313018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012543.xml
2022-11-23T01:33:43.6313249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6313517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6313635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6313913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6314048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6314053Z 
2022-11-23T01:33:43.6314111Z Running tests...
2022-11-23T01:33:43.6314302Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6314995Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.263s)
2022-11-23T01:33:43.6315001Z 
2022-11-23T01:33:43.6315189Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6315259Z Ran 1 test in 0.263s
2022-11-23T01:33:43.6315264Z 
2022-11-23T01:33:43.6315333Z OK (skipped=1)
2022-11-23T01:33:43.6315338Z 
2022-11-23T01:33:43.6315419Z Generating XML reports...
2022-11-23T01:33:43.6315757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012547.xml
2022-11-23T01:33:43.6315984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6316257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6316365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6316643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6316772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6316778Z 
2022-11-23T01:33:43.6316848Z Running tests...
2022-11-23T01:33:43.6317037Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6317417Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:33:43.6317425Z 
2022-11-23T01:33:43.6317615Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6317687Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6317692Z 
2022-11-23T01:33:43.6317759Z OK (skipped=1)
2022-11-23T01:33:43.6317764Z 
2022-11-23T01:33:43.6317872Z Generating XML reports...
2022-11-23T01:33:43.6318206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012549.xml
2022-11-23T01:33:43.6318432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6318701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6318818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6319094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6319254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6319259Z 
2022-11-23T01:33:43.6319326Z Running tests...
2022-11-23T01:33:43.6319517Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6319890Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:33:43.6319907Z 
2022-11-23T01:33:43.6320085Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6320156Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6320161Z 
2022-11-23T01:33:43.6320229Z OK (skipped=1)
2022-11-23T01:33:43.6320233Z 
2022-11-23T01:33:43.6320316Z Generating XML reports...
2022-11-23T01:33:43.6320652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml
2022-11-23T01:33:43.6320884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6321154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6321277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6321544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6321676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6321681Z 
2022-11-23T01:33:43.6321751Z Running tests...
2022-11-23T01:33:43.6321941Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6322174Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16749
2022-11-23T01:33:43.6322330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16750
2022-11-23T01:33:43.6322483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16751
2022-11-23T01:33:43.6322751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6322861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6323140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6323269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6323441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6323709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6323829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6324103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6324235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6324441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6324729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6324997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6325114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6325393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6325521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6325721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6326018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6326314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6326478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6326629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6326787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6326887Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:33:43.6326893Z 
2022-11-23T01:33:43.6327084Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6327157Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6327162Z 
2022-11-23T01:33:43.6327229Z OK (skipped=1)
2022-11-23T01:33:43.6327233Z 
2022-11-23T01:33:43.6327313Z Generating XML reports...
2022-11-23T01:33:43.6327650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012552.xml
2022-11-23T01:33:43.6327948Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6328209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6328327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6328606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6328737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6328744Z 
2022-11-23T01:33:43.6328814Z Running tests...
2022-11-23T01:33:43.6329006Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6329248Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16802
2022-11-23T01:33:43.6329400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16803
2022-11-23T01:33:43.6329542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16804
2022-11-23T01:33:43.6329809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6329927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6330206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6330338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6330513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6330781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6330900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6331209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6331330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6331504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6331796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6332060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6332207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6332480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6332611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6332784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6333077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6333361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6333523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6333681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6333837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6333939Z skip: Need at least 2 CUDA devices (2.072s)
2022-11-23T01:33:43.6333944Z 
2022-11-23T01:33:43.6334136Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6334207Z Ran 1 test in 2.072s
2022-11-23T01:33:43.6334214Z 
2022-11-23T01:33:43.6334283Z OK (skipped=1)
2022-11-23T01:33:43.6334287Z 
2022-11-23T01:33:43.6334357Z Generating XML reports...
2022-11-23T01:33:43.6334697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012556.xml
2022-11-23T01:33:43.6335016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6335283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6335401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6335682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6335815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6335821Z 
2022-11-23T01:33:43.6335889Z Running tests...
2022-11-23T01:33:43.6336080Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6336766Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:33:43.6336783Z 
2022-11-23T01:33:43.6336962Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6337036Z Ran 1 test in 0.254s
2022-11-23T01:33:43.6337041Z 
2022-11-23T01:33:43.6337107Z OK (skipped=1)
2022-11-23T01:33:43.6337111Z 
2022-11-23T01:33:43.6337192Z Generating XML reports...
2022-11-23T01:33:43.6337528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012600.xml
2022-11-23T01:33:43.6337802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6338078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6338198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6338474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6338595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6338600Z 
2022-11-23T01:33:43.6338705Z Running tests...
2022-11-23T01:33:43.6338897Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6339609Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T01:33:43.6339616Z 
2022-11-23T01:33:43.6339804Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6339876Z Ran 1 test in 0.255s
2022-11-23T01:33:43.6339881Z 
2022-11-23T01:33:43.6339949Z OK (skipped=1)
2022-11-23T01:33:43.6339954Z 
2022-11-23T01:33:43.6340035Z Generating XML reports...
2022-11-23T01:33:43.6340369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml
2022-11-23T01:33:43.6340598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6340859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6340977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6341256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6341392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6341398Z 
2022-11-23T01:33:43.6341519Z Running tests...
2022-11-23T01:33:43.6341710Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6342055Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.003s)
2022-11-23T01:33:43.6342061Z 
2022-11-23T01:33:43.6342250Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6342320Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6342325Z 
2022-11-23T01:33:43.6342384Z OK (skipped=1)
2022-11-23T01:33:43.6342389Z 
2022-11-23T01:33:43.6342471Z Generating XML reports...
2022-11-23T01:33:43.6342806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012604.xml
2022-11-23T01:33:43.6343035Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6343307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6343427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6343706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6343837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6343844Z 
2022-11-23T01:33:43.6343913Z Running tests...
2022-11-23T01:33:43.6344090Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6344352Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16885
2022-11-23T01:33:43.6344538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16886
2022-11-23T01:33:43.6344690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16887
2022-11-23T01:33:43.6344957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6345077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6345358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6345525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6345690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6345961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6346083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6346359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6346490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6346664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6346930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6347051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6347326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6347445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6347620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6347914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6348207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6348494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6348656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6348817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6348978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6349080Z skip: Need at least 2 CUDA devices (2.171s)
2022-11-23T01:33:43.6349085Z 
2022-11-23T01:33:43.6349271Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6349343Z Ran 1 test in 2.172s
2022-11-23T01:33:43.6349348Z 
2022-11-23T01:33:43.6349417Z OK (skipped=1)
2022-11-23T01:33:43.6349422Z 
2022-11-23T01:33:43.6349502Z Generating XML reports...
2022-11-23T01:33:43.6349839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012605.xml
2022-11-23T01:33:43.6350067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6350336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6350459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6350723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6350855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6350861Z 
2022-11-23T01:33:43.6350963Z Running tests...
2022-11-23T01:33:43.6351157Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6351392Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16938
2022-11-23T01:33:43.6351545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16939
2022-11-23T01:33:43.6351694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16940
2022-11-23T01:33:43.6351961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6352108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6352372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6352505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6352677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6352942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6353063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6353342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6353472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6353647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6353942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6354201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6354321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6354598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6354728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6354901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6355194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6355492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6355654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6355818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6355963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6356062Z skip: Need at least 3 CUDA devices (2.174s)
2022-11-23T01:33:43.6356068Z 
2022-11-23T01:33:43.6356260Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6356335Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6356339Z 
2022-11-23T01:33:43.6356410Z OK (skipped=1)
2022-11-23T01:33:43.6356415Z 
2022-11-23T01:33:43.6356497Z Generating XML reports...
2022-11-23T01:33:43.6356832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012609.xml
2022-11-23T01:33:43.6357061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6357319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6357500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6357779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6357912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6357916Z 
2022-11-23T01:33:43.6357985Z Running tests...
2022-11-23T01:33:43.6358175Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6358527Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.004s)
2022-11-23T01:33:43.6358559Z 
2022-11-23T01:33:43.6358746Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6358816Z Ran 1 test in 0.004s
2022-11-23T01:33:43.6358821Z 
2022-11-23T01:33:43.6358878Z OK (skipped=1)
2022-11-23T01:33:43.6358883Z 
2022-11-23T01:33:43.6358961Z Generating XML reports...
2022-11-23T01:33:43.6359299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012613.xml
2022-11-23T01:33:43.6359528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6359801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6359921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6360200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6360339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6360344Z 
2022-11-23T01:33:43.6360410Z Running tests...
2022-11-23T01:33:43.6360593Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6360931Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s)
2022-11-23T01:33:43.6360937Z 
2022-11-23T01:33:43.6361126Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6361199Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6361204Z 
2022-11-23T01:33:43.6361269Z OK (skipped=1)
2022-11-23T01:33:43.6361274Z 
2022-11-23T01:33:43.6361356Z Generating XML reports...
2022-11-23T01:33:43.6361691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012615.xml
2022-11-23T01:33:43.6361924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6362196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6362307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6362588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6362718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6362723Z 
2022-11-23T01:33:43.6362791Z Running tests...
2022-11-23T01:33:43.6362982Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6363208Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17011
2022-11-23T01:33:43.6363361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17012
2022-11-23T01:33:43.6363513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17013
2022-11-23T01:33:43.6363772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6363889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6364206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6364336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6364512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6364784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6364904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6365182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6365341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6365506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6365774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6365894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6366165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6366294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6366465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6366758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6367056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6367348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6367502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6367663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6367923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk67iy4k9
2022-11-23T01:33:43.6368104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw67dnoni
2022-11-23T01:33:43.6368293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk67iy4k9/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6368480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw67dnoni/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6368640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6368815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp408o1hq4
2022-11-23T01:33:43.6368995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp408o1hq4/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6369705Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6370386Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6371102Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6372000Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:33:43.6372199Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:33:43.6372358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6373255Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:33:43.6373425Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:33:43.6373592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6374479Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:33:43.6374645Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-11-23T01:33:43.6374808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6375591Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6376271Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6376947Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6377673Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6378351Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6379019Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6379728Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6380403Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6381076Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6381743Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6382413Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6383082Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6383754Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6384422Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6385115Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:33:43.6385181Z ok (2.176s)
2022-11-23T01:33:43.6385187Z 
2022-11-23T01:33:43.6385382Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6385455Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6385487Z 
2022-11-23T01:33:43.6385544Z OK
2022-11-23T01:33:43.6385549Z 
2022-11-23T01:33:43.6385632Z Generating XML reports...
2022-11-23T01:33:43.6385979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012616.xml
2022-11-23T01:33:43.6386210Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6386487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6386598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6386876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6387008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6387013Z 
2022-11-23T01:33:43.6387081Z Running tests...
2022-11-23T01:33:43.6387272Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6387578Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.004s)
2022-11-23T01:33:43.6387583Z 
2022-11-23T01:33:43.6387775Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6387847Z Ran 1 test in 0.004s
2022-11-23T01:33:43.6387852Z 
2022-11-23T01:33:43.6387922Z OK (skipped=1)
2022-11-23T01:33:43.6387928Z 
2022-11-23T01:33:43.6387998Z Generating XML reports...
2022-11-23T01:33:43.6388334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012620.xml
2022-11-23T01:33:43.6388562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6388835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6388955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6389235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6389367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6389372Z 
2022-11-23T01:33:43.6389441Z Running tests...
2022-11-23T01:33:43.6389623Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6389947Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.003s)
2022-11-23T01:33:43.6389952Z 
2022-11-23T01:33:43.6390141Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6390214Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6390219Z 
2022-11-23T01:33:43.6390286Z OK (skipped=1)
2022-11-23T01:33:43.6390290Z 
2022-11-23T01:33:43.6390368Z Generating XML reports...
2022-11-23T01:33:43.6390698Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012622.xml
2022-11-23T01:33:43.6390931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6391203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6391344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6391620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6391752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6391756Z 
2022-11-23T01:33:43.6391825Z Running tests...
2022-11-23T01:33:43.6392016Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6392704Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:33:43.6392737Z 
2022-11-23T01:33:43.6392927Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6393003Z Ran 1 test in 0.253s
2022-11-23T01:33:43.6393007Z 
2022-11-23T01:33:43.6393076Z OK (skipped=1)
2022-11-23T01:33:43.6393081Z 
2022-11-23T01:33:43.6393162Z Generating XML reports...
2022-11-23T01:33:43.6393489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012624.xml
2022-11-23T01:33:43.6393719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6393987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6394109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6394386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6394519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6394524Z 
2022-11-23T01:33:43.6394595Z Running tests...
2022-11-23T01:33:43.6394786Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6395478Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T01:33:43.6395484Z 
2022-11-23T01:33:43.6395674Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6395737Z Ran 1 test in 0.252s
2022-11-23T01:33:43.6395742Z 
2022-11-23T01:33:43.6395810Z OK (skipped=1)
2022-11-23T01:33:43.6395815Z 
2022-11-23T01:33:43.6395893Z Generating XML reports...
2022-11-23T01:33:43.6396227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml
2022-11-23T01:33:43.6396460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6396730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6396850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6397129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6397260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6397268Z 
2022-11-23T01:33:43.6397326Z Running tests...
2022-11-23T01:33:43.6397515Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6397767Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17104
2022-11-23T01:33:43.6397950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17105
2022-11-23T01:33:43.6398105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17106
2022-11-23T01:33:43.6398378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6398497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6398777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6398899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6399100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6399369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6399486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6399763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6399896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6400072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6400336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6400457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6400723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6400852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6401025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6401323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6401618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6401906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6402069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6402230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6402393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6402483Z skip: Need at least 3 CUDA devices (2.175s)
2022-11-23T01:33:43.6402488Z 
2022-11-23T01:33:43.6402682Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6402755Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6402760Z 
2022-11-23T01:33:43.6402826Z OK (skipped=1)
2022-11-23T01:33:43.6402831Z 
2022-11-23T01:33:43.6402912Z Generating XML reports...
2022-11-23T01:33:43.6403251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012627.xml
2022-11-23T01:33:43.6403481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6403752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6403863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6404138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6404268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6404273Z 
2022-11-23T01:33:43.6404342Z Running tests...
2022-11-23T01:33:43.6404561Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6404805Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17157
2022-11-23T01:33:43.6404958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17158
2022-11-23T01:33:43.6405111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17159
2022-11-23T01:33:43.6405381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6405516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6405794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6405924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6406100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6406370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6406489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6406765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6406896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6407070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6407353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6407620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6407808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6408086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6408217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6408392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6408687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6408981Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6409144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6409293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6409447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6409547Z skip: Need at least 3 CUDA devices (2.079s)
2022-11-23T01:33:43.6409552Z 
2022-11-23T01:33:43.6409745Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6409818Z Ran 1 test in 2.079s
2022-11-23T01:33:43.6409823Z 
2022-11-23T01:33:43.6409892Z OK (skipped=1)
2022-11-23T01:33:43.6409896Z 
2022-11-23T01:33:43.6409977Z Generating XML reports...
2022-11-23T01:33:43.6410319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012631.xml
2022-11-23T01:33:43.6410537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6410806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6410925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6411245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6411376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6411381Z 
2022-11-23T01:33:43.6411451Z Running tests...
2022-11-23T01:33:43.6411642Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6412335Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:33:43.6412368Z 
2022-11-23T01:33:43.6412558Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6412630Z Ran 1 test in 0.255s
2022-11-23T01:33:43.6412637Z 
2022-11-23T01:33:43.6412694Z OK (skipped=1)
2022-11-23T01:33:43.6412698Z 
2022-11-23T01:33:43.6412777Z Generating XML reports...
2022-11-23T01:33:43.6413109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012635.xml
2022-11-23T01:33:43.6413334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6413605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6413727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6414006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6414137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6414142Z 
2022-11-23T01:33:43.6414209Z Running tests...
2022-11-23T01:33:43.6414391Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6414627Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17220
2022-11-23T01:33:43.6414780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17221
2022-11-23T01:33:43.6415022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17222
2022-11-23T01:33:43.6415297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6415421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6415700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6415830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6415994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6416264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6416384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6416660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6416792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6416967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6417242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6417361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6417684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6417805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6417977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6418275Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6418569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6418858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6419056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6419219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6419380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6419483Z skip: Need at least 3 CUDA devices (2.174s)
2022-11-23T01:33:43.6419488Z 
2022-11-23T01:33:43.6419670Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6419743Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6419748Z 
2022-11-23T01:33:43.6419815Z OK (skipped=1)
2022-11-23T01:33:43.6419819Z 
2022-11-23T01:33:43.6419900Z Generating XML reports...
2022-11-23T01:33:43.6420238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012637.xml
2022-11-23T01:33:43.6420470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6420740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6420859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6421138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6421258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6421263Z 
2022-11-23T01:33:43.6421331Z Running tests...
2022-11-23T01:33:43.6421524Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6421793Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17273
2022-11-23T01:33:43.6421952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17274
2022-11-23T01:33:43.6422104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17275
2022-11-23T01:33:43.6422372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6422493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6422761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6422893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6423068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6423339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6423461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6423740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6423873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6424048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6424343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6424453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6424731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6424860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6425033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6425353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6425645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6425935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6426098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6426254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6426405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6426505Z skip: Need at least 2 CUDA devices (2.175s)
2022-11-23T01:33:43.6426510Z 
2022-11-23T01:33:43.6426704Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6426780Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6426785Z 
2022-11-23T01:33:43.6426854Z OK (skipped=1)
2022-11-23T01:33:43.6426858Z 
2022-11-23T01:33:43.6426939Z Generating XML reports...
2022-11-23T01:33:43.6427277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012641.xml
2022-11-23T01:33:43.6427506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6427766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6427885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6428160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6428289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6428294Z 
2022-11-23T01:33:43.6428365Z Running tests...
2022-11-23T01:33:43.6428555Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6428825Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17326
2022-11-23T01:33:43.6428981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17327
2022-11-23T01:33:43.6429133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17328
2022-11-23T01:33:43.6429393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6429512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6429789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6429919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6430096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6430366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6430486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6430790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6430922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6431087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6431353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6431472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6431781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6431912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6432084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6432386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6432683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6432974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6433126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6433282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6433445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6433545Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:33:43.6433550Z 
2022-11-23T01:33:43.6433741Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6433814Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6433821Z 
2022-11-23T01:33:43.6433892Z OK (skipped=1)
2022-11-23T01:33:43.6433896Z 
2022-11-23T01:33:43.6433979Z Generating XML reports...
2022-11-23T01:33:43.6434305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012645.xml
2022-11-23T01:33:43.6434532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6434801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6434925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6435202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6435331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6435336Z 
2022-11-23T01:33:43.6435406Z Running tests...
2022-11-23T01:33:43.6435598Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6435913Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17379
2022-11-23T01:33:43.6436057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17380
2022-11-23T01:33:43.6436208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17381
2022-11-23T01:33:43.6436479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6436601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6436883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6437040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6437213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6437482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6437603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6437869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6438001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6438208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6438476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6438594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6438872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6439002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6439173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6439468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6439750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6440045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6440206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6440363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6440525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6440625Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:33:43.6440630Z 
2022-11-23T01:33:43.6440823Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6440895Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6440900Z 
2022-11-23T01:33:43.6440957Z OK (skipped=1)
2022-11-23T01:33:43.6440972Z 
2022-11-23T01:33:43.6441041Z Generating XML reports...
2022-11-23T01:33:43.6441379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012649.xml
2022-11-23T01:33:43.6441613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6441884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6442004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6442281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6442412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6442417Z 
2022-11-23T01:33:43.6442487Z Running tests...
2022-11-23T01:33:43.6442670Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6442985Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17432
2022-11-23T01:33:43.6443140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17433
2022-11-23T01:33:43.6443291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17434
2022-11-23T01:33:43.6443589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6443709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6443985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6444118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6444292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6444550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6444699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6444976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6445106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6445284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6445579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6445847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6445966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6446241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6446363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6446537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6446836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6447137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6447304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6447467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6447622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6447793Z skip: Need at least 2 CUDA devices (2.077s)
2022-11-23T01:33:43.6447801Z 
2022-11-23T01:33:43.6447999Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6448060Z Ran 1 test in 2.077s
2022-11-23T01:33:43.6448064Z 
2022-11-23T01:33:43.6448135Z OK (skipped=1)
2022-11-23T01:33:43.6448139Z 
2022-11-23T01:33:43.6448222Z Generating XML reports...
2022-11-23T01:33:43.6448564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012653.xml
2022-11-23T01:33:43.6448797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6449068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6449189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6449471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6449594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6449609Z 
2022-11-23T01:33:43.6449668Z Running tests...
2022-11-23T01:33:43.6449861Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6450215Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17485
2022-11-23T01:33:43.6450372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17486
2022-11-23T01:33:43.6450521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17487
2022-11-23T01:33:43.6450793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6450913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6451195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6451344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6451520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6451795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6451916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6452194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6452324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6452498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6452762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6452884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6453149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6453278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6453453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6453751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6454044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6454337Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6454501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6454663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6454823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6455020Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:33:43.6455027Z 
2022-11-23T01:33:43.6455222Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6455295Z Ran 1 test in 2.077s
2022-11-23T01:33:43.6455300Z 
2022-11-23T01:33:43.6455371Z OK (skipped=1)
2022-11-23T01:33:43.6455376Z 
2022-11-23T01:33:43.6455457Z Generating XML reports...
2022-11-23T01:33:43.6455797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012657.xml
2022-11-23T01:33:43.6456024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6456297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6456406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6456684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6456875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6456880Z 
2022-11-23T01:33:43.6456950Z Running tests...
2022-11-23T01:33:43.6457144Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6457459Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17538
2022-11-23T01:33:43.6457614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17539
2022-11-23T01:33:43.6457835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17540
2022-11-23T01:33:43.6458107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6458216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6458501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6458633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6458808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6459077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6459196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6459473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6459608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6459783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6460069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6460341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6460460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6460739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6460869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6461040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6461337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6461634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6461798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6461949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6462107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6462208Z skip: Need at least 2 CUDA devices (2.171s)
2022-11-23T01:33:43.6462213Z 
2022-11-23T01:33:43.6462407Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6462478Z Ran 1 test in 2.171s
2022-11-23T01:33:43.6462483Z 
2022-11-23T01:33:43.6462555Z OK (skipped=1)
2022-11-23T01:33:43.6462560Z 
2022-11-23T01:33:43.6462639Z Generating XML reports...
2022-11-23T01:33:43.6462977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012700.xml
2022-11-23T01:33:43.6463199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6463495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6463615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6463892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6464023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6464028Z 
2022-11-23T01:33:43.6464098Z Running tests...
2022-11-23T01:33:43.6464290Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6464632Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17591
2022-11-23T01:33:43.6464787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17592
2022-11-23T01:33:43.6464930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17593
2022-11-23T01:33:43.6465202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6465323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6465601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6465732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6465907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6466174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6466292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6466571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6466692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6466868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6467164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6467431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6467554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6467828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6467961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6468138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6468432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6468717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6468878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6469033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6469197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6469300Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:33:43.6469305Z 
2022-11-23T01:33:43.6469493Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6469563Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6469568Z 
2022-11-23T01:33:43.6469668Z OK (skipped=1)
2022-11-23T01:33:43.6469673Z 
2022-11-23T01:33:43.6469757Z Generating XML reports...
2022-11-23T01:33:43.6470087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012704.xml
2022-11-23T01:33:43.6470318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6470588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6470707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6471009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6471142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6471146Z 
2022-11-23T01:33:43.6471215Z Running tests...
2022-11-23T01:33:43.6471410Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6471710Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17644
2022-11-23T01:33:43.6471867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17645
2022-11-23T01:33:43.6472021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17646
2022-11-23T01:33:43.6472287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6472410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6472684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6472813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6472987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6473256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6473363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6473640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6473771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6473951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6474244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6474513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6474634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6474910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6475110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6475357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6475709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6476039Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6476327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6476532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6476756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6476889Z skip: Need at least 2 CUDA devices (2.077s)
2022-11-23T01:33:43.6476894Z 
2022-11-23T01:33:43.6477120Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6477182Z Ran 1 test in 2.077s
2022-11-23T01:33:43.6477187Z 
2022-11-23T01:33:43.6477285Z OK (skipped=1)
2022-11-23T01:33:43.6477289Z 
2022-11-23T01:33:43.6477400Z Generating XML reports...
2022-11-23T01:33:43.6477908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012708.xml
2022-11-23T01:33:43.6490523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6490938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6491072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6491376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6491515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6491522Z 
2022-11-23T01:33:43.6491599Z Running tests...
2022-11-23T01:33:43.6491805Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6492129Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17697
2022-11-23T01:33:43.6492284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17698
2022-11-23T01:33:43.6492443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17699
2022-11-23T01:33:43.6492725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6492852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6493134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6493271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6493451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6493728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6493855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6494130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6494268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6494451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6494728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6495027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6495315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6495450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6495632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6495923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6496227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6496672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6496859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6497024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6497189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6497297Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:33:43.6497303Z 
2022-11-23T01:33:43.6497545Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6497624Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6497630Z 
2022-11-23T01:33:43.6497691Z OK (skipped=1)
2022-11-23T01:33:43.6497711Z 
2022-11-23T01:33:43.6497783Z Generating XML reports...
2022-11-23T01:33:43.6498134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012712.xml
2022-11-23T01:33:43.6498369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6498647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6498774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6499060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6499196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6499204Z 
2022-11-23T01:33:43.6499279Z Running tests...
2022-11-23T01:33:43.6499463Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6499784Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17750
2022-11-23T01:33:43.6499945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17751
2022-11-23T01:33:43.6500101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17752
2022-11-23T01:33:43.6500375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6500505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6500889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6501032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6501211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6501474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6501605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6501888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6502027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6502207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6502480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6502610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6502893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6503016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6503228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6503533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6503834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6504131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6504302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6504498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6504664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6504771Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:33:43.6504776Z 
2022-11-23T01:33:43.6504965Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6505046Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6505051Z 
2022-11-23T01:33:43.6505124Z OK (skipped=1)
2022-11-23T01:33:43.6505129Z 
2022-11-23T01:33:43.6505214Z Generating XML reports...
2022-11-23T01:33:43.6505560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012716.xml
2022-11-23T01:33:43.6505796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6506077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6506206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6506494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6506622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6506627Z 
2022-11-23T01:33:43.6506706Z Running tests...
2022-11-23T01:33:43.6506904Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6507181Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17803
2022-11-23T01:33:43.6507339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17804
2022-11-23T01:33:43.6507495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17805
2022-11-23T01:33:43.6507777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6507903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6508200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6508326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6508510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6508787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6508910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6509194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6509331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6509509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6509784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6509922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6510210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6510344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6510523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6510824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6511144Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6511465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6511653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6511815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6511965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6512090Z skip: Need at least 2 CUDA devices (2.072s)
2022-11-23T01:33:43.6512095Z 
2022-11-23T01:33:43.6512293Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6512372Z Ran 1 test in 2.073s
2022-11-23T01:33:43.6512378Z 
2022-11-23T01:33:43.6512450Z OK (skipped=1)
2022-11-23T01:33:43.6512455Z 
2022-11-23T01:33:43.6512542Z Generating XML reports...
2022-11-23T01:33:43.6512887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012720.xml
2022-11-23T01:33:43.6513151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6513499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6513645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6519643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6519826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6519834Z 
2022-11-23T01:33:43.6519939Z Running tests...
2022-11-23T01:33:43.6520229Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6520617Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17856
2022-11-23T01:33:43.6520867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17857
2022-11-23T01:33:43.6521105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17858
2022-11-23T01:33:43.6521524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6521712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6522152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6522357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6522610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6523034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6523224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6523697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6524011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6524305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6524804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6525271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6525468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6525946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6526222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6526509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6527003Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6527515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6527847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6528115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6528372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6528529Z skip: Need at least 2 CUDA devices (2.173s)
2022-11-23T01:33:43.6528544Z 
2022-11-23T01:33:43.6528858Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6528973Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6528981Z 
2022-11-23T01:33:43.6529091Z OK (skipped=1)
2022-11-23T01:33:43.6529099Z 
2022-11-23T01:33:43.6529229Z Generating XML reports...
2022-11-23T01:33:43.6529811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012724.xml
2022-11-23T01:33:43.6530196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6530663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6551479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6551838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6551975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6551981Z 
2022-11-23T01:33:43.6552047Z Running tests...
2022-11-23T01:33:43.6552239Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6552559Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:33:43.6552564Z 
2022-11-23T01:33:43.6552748Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6552809Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6552814Z 
2022-11-23T01:33:43.6552878Z OK (skipped=1)
2022-11-23T01:33:43.6552883Z 
2022-11-23T01:33:43.6552961Z Generating XML reports...
2022-11-23T01:33:43.6553299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012728.xml
2022-11-23T01:33:43.6553524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6553792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6553907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6554282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6554405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6554417Z 
2022-11-23T01:33:43.6554476Z Running tests...
2022-11-23T01:33:43.6554664Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6554888Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17919
2022-11-23T01:33:43.6555037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17920
2022-11-23T01:33:43.6555249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17921
2022-11-23T01:33:43.6555517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6555632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6555910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6556032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6556203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6556469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6556584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6556859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6556985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6557155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6557422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6557537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6557804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6557932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6558100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6558393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6558687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6558978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6559137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6559294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6559448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6559539Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:33:43.6559545Z 
2022-11-23T01:33:43.6559735Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6559803Z Ran 1 test in 2.076s
2022-11-23T01:33:43.6559807Z 
2022-11-23T01:33:43.6559870Z OK (skipped=1)
2022-11-23T01:33:43.6559877Z 
2022-11-23T01:33:43.6559953Z Generating XML reports...
2022-11-23T01:33:43.6560289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml
2022-11-23T01:33:43.6560515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6560812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6560924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6561197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6561324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6561329Z 
2022-11-23T01:33:43.6561394Z Running tests...
2022-11-23T01:33:43.6561581Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6561846Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17972
2022-11-23T01:33:43.6561995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17973
2022-11-23T01:33:43.6562140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17974
2022-11-23T01:33:43.6562409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6562518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6562793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6562920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6563090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6563357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6563472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6563746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6563876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6564039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6564304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6564422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6564697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6564825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6564993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6565287Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6565581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6565868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6566020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6566176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6566330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6566430Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:33:43.6566435Z 
2022-11-23T01:33:43.6566623Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6566691Z Ran 1 test in 2.176s
2022-11-23T01:33:43.6566696Z 
2022-11-23T01:33:43.6566760Z OK (skipped=1)
2022-11-23T01:33:43.6566765Z 
2022-11-23T01:33:43.6566841Z Generating XML reports...
2022-11-23T01:33:43.6567204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012733.xml
2022-11-23T01:33:43.6567425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6567779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6567901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6568178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6568339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6568343Z 
2022-11-23T01:33:43.6568408Z Running tests...
2022-11-23T01:33:43.6568597Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6568828Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18025
2022-11-23T01:33:43.6568978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18026
2022-11-23T01:33:43.6569121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18027
2022-11-23T01:33:43.6569388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6569501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6569783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6569917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6570097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6570370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6570493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6570760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6570895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6571072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6571343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6571467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6571746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6571879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6572058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6572357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6572639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6572930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6573099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6573287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrgxhdrm
2022-11-23T01:33:43.6573487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrgxhdrm/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6573678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6573845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6574028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyxyns22
2022-11-23T01:33:43.6574210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79f8xxbj
2022-11-23T01:33:43.6574391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyxyns22/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6574577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79f8xxbj/_remote_module_non_scriptable.py
2022-11-23T01:33:43.6574776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6575133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6575298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:33:43.6575371Z ok (2.277s)
2022-11-23T01:33:43.6575377Z 
2022-11-23T01:33:43.6575581Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6575655Z Ran 1 test in 2.277s
2022-11-23T01:33:43.6575660Z 
2022-11-23T01:33:43.6575707Z OK
2022-11-23T01:33:43.6575711Z 
2022-11-23T01:33:43.6575797Z Generating XML reports...
2022-11-23T01:33:43.6576144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012737.xml
2022-11-23T01:33:43.6576378Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6576655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6576779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6577060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6577198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6577203Z 
2022-11-23T01:33:43.6577277Z Running tests...
2022-11-23T01:33:43.6577458Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6577692Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18087
2022-11-23T01:33:43.6577846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18088
2022-11-23T01:33:43.6578002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18089
2022-11-23T01:33:43.6578278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6578400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6578684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6578818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6578982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6579256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6579379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6579663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6579800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6579977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6580245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6580413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6580693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6580813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6580988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6581289Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6581622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6581919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6582084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6582247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6582409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6582504Z skip: CUDA is not available. (2.178s)
2022-11-23T01:33:43.6582508Z 
2022-11-23T01:33:43.6582691Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6582764Z Ran 1 test in 2.179s
2022-11-23T01:33:43.6582769Z 
2022-11-23T01:33:43.6582839Z OK (skipped=1)
2022-11-23T01:33:43.6582846Z 
2022-11-23T01:33:43.6582930Z Generating XML reports...
2022-11-23T01:33:43.6583273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012741.xml
2022-11-23T01:33:43.6583505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6583782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6583903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6584171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6584308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6584313Z 
2022-11-23T01:33:43.6584384Z Running tests...
2022-11-23T01:33:43.6584575Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6584925Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.002s)
2022-11-23T01:33:43.6584930Z 
2022-11-23T01:33:43.6585116Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6585191Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6585196Z 
2022-11-23T01:33:43.6585267Z OK (skipped=1)
2022-11-23T01:33:43.6585273Z 
2022-11-23T01:33:43.6585358Z Generating XML reports...
2022-11-23T01:33:43.6585687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012745.xml
2022-11-23T01:33:43.6585919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6586193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6586313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6586596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6586730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6586735Z 
2022-11-23T01:33:43.6586806Z Running tests...
2022-11-23T01:33:43.6586998Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6587367Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s)
2022-11-23T01:33:43.6587373Z 
2022-11-23T01:33:43.6587552Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6587626Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6587631Z 
2022-11-23T01:33:43.6587699Z OK (skipped=1)
2022-11-23T01:33:43.6587704Z 
2022-11-23T01:33:43.6587787Z Generating XML reports...
2022-11-23T01:33:43.6588126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012747.xml
2022-11-23T01:33:43.6588384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6588657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6588782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6589062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6589181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6589186Z 
2022-11-23T01:33:43.6589257Z Running tests...
2022-11-23T01:33:43.6589449Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6589821Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo', 'ucc'} (0.001s)
2022-11-23T01:33:43.6589829Z 
2022-11-23T01:33:43.6590019Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6590093Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6590098Z 
2022-11-23T01:33:43.6590168Z OK (skipped=1)
2022-11-23T01:33:43.6590172Z 
2022-11-23T01:33:43.6590259Z Generating XML reports...
2022-11-23T01:33:43.6590600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012748.xml
2022-11-23T01:33:43.6590820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6591095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6591217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6591497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6591633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6591637Z 
2022-11-23T01:33:43.6591708Z Running tests...
2022-11-23T01:33:43.6591903Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6592260Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:33:43.6592265Z 
2022-11-23T01:33:43.6592458Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6592518Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6592523Z 
2022-11-23T01:33:43.6592592Z OK (skipped=1)
2022-11-23T01:33:43.6592596Z 
2022-11-23T01:33:43.6592680Z Generating XML reports...
2022-11-23T01:33:43.6593018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml
2022-11-23T01:33:43.6593248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6593523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6593648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6593955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6594093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6594097Z 
2022-11-23T01:33:43.6594156Z Running tests...
2022-11-23T01:33:43.6594347Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6594662Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'ucc', 'gloo'} (0.002s)
2022-11-23T01:33:43.6594668Z 
2022-11-23T01:33:43.6594858Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6594961Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6594965Z 
2022-11-23T01:33:43.6595035Z OK (skipped=1)
2022-11-23T01:33:43.6595039Z 
2022-11-23T01:33:43.6595121Z Generating XML reports...
2022-11-23T01:33:43.6595462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012751.xml
2022-11-23T01:33:43.6595681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6595955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6596078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6596358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6596489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6596494Z 
2022-11-23T01:33:43.6596569Z Running tests...
2022-11-23T01:33:43.6596760Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6596994Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18190
2022-11-23T01:33:43.6597151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18191
2022-11-23T01:33:43.6597293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18192
2022-11-23T01:33:43.6597565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6597687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6597966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6598099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6598278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6598548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6598670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6598949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6599070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6599245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6599514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6599634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6599911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6600046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6600220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6600547Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6600835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6601130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6601340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6601537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6601744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6601849Z skip: Need at least 2 CUDA devices (2.171s)
2022-11-23T01:33:43.6601854Z 
2022-11-23T01:33:43.6602053Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6602129Z Ran 1 test in 2.172s
2022-11-23T01:33:43.6602133Z 
2022-11-23T01:33:43.6602206Z OK (skipped=1)
2022-11-23T01:33:43.6602211Z 
2022-11-23T01:33:43.6602282Z Generating XML reports...
2022-11-23T01:33:43.6602624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012753.xml
2022-11-23T01:33:43.6602856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6603128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6603250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6603533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6603666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6603671Z 
2022-11-23T01:33:43.6603744Z Running tests...
2022-11-23T01:33:43.6603940Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6604641Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:33:43.6604661Z 
2022-11-23T01:33:43.6604838Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6604914Z Ran 1 test in 0.255s
2022-11-23T01:33:43.6604919Z 
2022-11-23T01:33:43.6604990Z OK (skipped=1)
2022-11-23T01:33:43.6604995Z 
2022-11-23T01:33:43.6605080Z Generating XML reports...
2022-11-23T01:33:43.6605421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml
2022-11-23T01:33:43.6605654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6605928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6606050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6606331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6606453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6606458Z 
2022-11-23T01:33:43.6606529Z Running tests...
2022-11-23T01:33:43.6606724Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6607062Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:33:43.6607067Z 
2022-11-23T01:33:43.6607259Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6607361Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6607366Z 
2022-11-23T01:33:43.6607437Z OK (skipped=1)
2022-11-23T01:33:43.6607441Z 
2022-11-23T01:33:43.6607524Z Generating XML reports...
2022-11-23T01:33:43.6607930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012759.xml
2022-11-23T01:33:43.6608151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6608423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6608587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6608865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6608999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6609004Z 
2022-11-23T01:33:43.6609080Z Running tests...
2022-11-23T01:33:43.6609273Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6609605Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.002s)
2022-11-23T01:33:43.6609611Z 
2022-11-23T01:33:43.6609802Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6609862Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6609867Z 
2022-11-23T01:33:43.6609935Z OK (skipped=1)
2022-11-23T01:33:43.6609939Z 
2022-11-23T01:33:43.6610025Z Generating XML reports...
2022-11-23T01:33:43.6610361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012800.xml
2022-11-23T01:33:43.6610594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6610868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6610991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6611271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6611392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6611410Z 
2022-11-23T01:33:43.6611468Z Running tests...
2022-11-23T01:33:43.6611658Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6611899Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18273
2022-11-23T01:33:43.6612059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18274
2022-11-23T01:33:43.6612213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18275
2022-11-23T01:33:43.6612487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6612610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6612889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6613010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6613188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6613460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6613584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6613862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6614026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6614204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6614475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6614583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6614965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6615110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6615340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6615643Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6615942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6616230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6616396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6616562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6616710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6616814Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:33:43.6616822Z 
2022-11-23T01:33:43.6617019Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6617094Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6617099Z 
2022-11-23T01:33:43.6617169Z OK (skipped=1)
2022-11-23T01:33:43.6617173Z 
2022-11-23T01:33:43.6617256Z Generating XML reports...
2022-11-23T01:33:43.6617597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012802.xml
2022-11-23T01:33:43.6617830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6618105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6618213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6618493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6618630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6618634Z 
2022-11-23T01:33:43.6618707Z Running tests...
2022-11-23T01:33:43.6618904Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6619610Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T01:33:43.6619616Z 
2022-11-23T01:33:43.6619804Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6619878Z Ran 1 test in 0.252s
2022-11-23T01:33:43.6619883Z 
2022-11-23T01:33:43.6619953Z OK (skipped=1)
2022-11-23T01:33:43.6619957Z 
2022-11-23T01:33:43.6620039Z Generating XML reports...
2022-11-23T01:33:43.6620369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012806.xml
2022-11-23T01:33:43.6620600Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6620910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6621035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6621316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6621450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6621455Z 
2022-11-23T01:33:43.6621527Z Running tests...
2022-11-23T01:33:43.6621720Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6622061Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'gloo', 'nccl'} (0.002s)
2022-11-23T01:33:43.6622100Z 
2022-11-23T01:33:43.6622276Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6622351Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6622356Z 
2022-11-23T01:33:43.6622427Z OK (skipped=1)
2022-11-23T01:33:43.6622432Z 
2022-11-23T01:33:43.6622518Z Generating XML reports...
2022-11-23T01:33:43.6622863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012808.xml
2022-11-23T01:33:43.6623090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6623363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6623486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6623752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6623888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6623893Z 
2022-11-23T01:33:43.6623964Z Running tests...
2022-11-23T01:33:43.6624159Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6624863Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s)
2022-11-23T01:33:43.6624869Z 
2022-11-23T01:33:43.6625063Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6625136Z Ran 1 test in 0.259s
2022-11-23T01:33:43.6625141Z 
2022-11-23T01:33:43.6625212Z OK (skipped=1)
2022-11-23T01:33:43.6625217Z 
2022-11-23T01:33:43.6625302Z Generating XML reports...
2022-11-23T01:33:43.6625639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012809.xml
2022-11-23T01:33:43.6625858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6626137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6626260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6626540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6626675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6626680Z 
2022-11-23T01:33:43.6626750Z Running tests...
2022-11-23T01:33:43.6626944Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6627186Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18356
2022-11-23T01:33:43.6627344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18357
2022-11-23T01:33:43.6627486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18358
2022-11-23T01:33:43.6627820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6627943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6628225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6628359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6628535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6628835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6628958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6629235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6629358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6629531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6629830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6630101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6630223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6630506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6630638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6630811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6631093Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6631395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6631559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6631723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6631884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6631990Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:33:43.6631995Z 
2022-11-23T01:33:43.6632188Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6632261Z Ran 1 test in 2.076s
2022-11-23T01:33:43.6632266Z 
2022-11-23T01:33:43.6632337Z OK (skipped=1)
2022-11-23T01:33:43.6632343Z 
2022-11-23T01:33:43.6632413Z Generating XML reports...
2022-11-23T01:33:43.6632760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012811.xml
2022-11-23T01:33:43.6632991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6633264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6633388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6633667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6633803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6633808Z 
2022-11-23T01:33:43.6633879Z Running tests...
2022-11-23T01:33:43.6634075Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6634325Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18409
2022-11-23T01:33:43.6634483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18410
2022-11-23T01:33:43.6634637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18411
2022-11-23T01:33:43.6634912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6635033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6635314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6635475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6635649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6635910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6636033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6636311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6636444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6636621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6636889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6637013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6637291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6637423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6637586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6637883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6638179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6638469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6638635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6638792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6638954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6639057Z skip: Need at least 2 CUDA devices (2.075s)
2022-11-23T01:33:43.6639064Z 
2022-11-23T01:33:43.6639260Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6639321Z Ran 1 test in 2.075s
2022-11-23T01:33:43.6639326Z 
2022-11-23T01:33:43.6639395Z OK (skipped=1)
2022-11-23T01:33:43.6639399Z 
2022-11-23T01:33:43.6639481Z Generating XML reports...
2022-11-23T01:33:43.6639819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml
2022-11-23T01:33:43.6640051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6640326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6640448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6640727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6640887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6640892Z 
2022-11-23T01:33:43.6640952Z Running tests...
2022-11-23T01:33:43.6641147Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6641391Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18462
2022-11-23T01:33:43.6641548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18463
2022-11-23T01:33:43.6641700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18464
2022-11-23T01:33:43.6642000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6642121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6642402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6642525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6642702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6642974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6643096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6643373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6643504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6643681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6643980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6644253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6644362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6644636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6644769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6644942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6645240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6645535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6645697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6645862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6646024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6646113Z skip: Need at least 2 CUDA devices (2.078s)
2022-11-23T01:33:43.6646118Z 
2022-11-23T01:33:43.6646311Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6646385Z Ran 1 test in 2.078s
2022-11-23T01:33:43.6646390Z 
2022-11-23T01:33:43.6646460Z OK (skipped=1)
2022-11-23T01:33:43.6646464Z 
2022-11-23T01:33:43.6646550Z Generating XML reports...
2022-11-23T01:33:43.6646892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012819.xml
2022-11-23T01:33:43.6647127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6647425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6647537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6647922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6648058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6648063Z 
2022-11-23T01:33:43.6648134Z Running tests...
2022-11-23T01:33:43.6648330Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6649030Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s)
2022-11-23T01:33:43.6649085Z 
2022-11-23T01:33:43.6649283Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6649359Z Ran 1 test in 0.259s
2022-11-23T01:33:43.6649364Z 
2022-11-23T01:33:43.6649434Z OK (skipped=1)
2022-11-23T01:33:43.6649439Z 
2022-11-23T01:33:43.6649522Z Generating XML reports...
2022-11-23T01:33:43.6649850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012823.xml
2022-11-23T01:33:43.6650082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6650353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6650479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6650758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6650893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6650898Z 
2022-11-23T01:33:43.6650969Z Running tests...
2022-11-23T01:33:43.6651165Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6651844Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.260s)
2022-11-23T01:33:43.6651852Z 
2022-11-23T01:33:43.6652041Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6652100Z Ran 1 test in 0.260s
2022-11-23T01:33:43.6652105Z 
2022-11-23T01:33:43.6652176Z OK (skipped=1)
2022-11-23T01:33:43.6652180Z 
2022-11-23T01:33:43.6652264Z Generating XML reports...
2022-11-23T01:33:43.6652607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012825.xml
2022-11-23T01:33:43.6652837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6653108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6653234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6653514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6653650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6653655Z 
2022-11-23T01:33:43.6653713Z Running tests...
2022-11-23T01:33:43.6653903Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6654634Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T01:33:43.6654641Z 
2022-11-23T01:33:43.6654831Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6655031Z Ran 1 test in 0.255s
2022-11-23T01:33:43.6655037Z 
2022-11-23T01:33:43.6655109Z OK (skipped=1)
2022-11-23T01:33:43.6655113Z 
2022-11-23T01:33:43.6655196Z Generating XML reports...
2022-11-23T01:33:43.6655590Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml
2022-11-23T01:33:43.6655821Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6656095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6656208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6656489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6656623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6656628Z 
2022-11-23T01:33:43.6656701Z Running tests...
2022-11-23T01:33:43.6656893Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6657317Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.003s)
2022-11-23T01:33:43.6657325Z 
2022-11-23T01:33:43.6657515Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6657588Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6657593Z 
2022-11-23T01:33:43.6657662Z OK (skipped=1)
2022-11-23T01:33:43.6657667Z 
2022-11-23T01:33:43.6657738Z Generating XML reports...
2022-11-23T01:33:43.6658075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012829.xml
2022-11-23T01:33:43.6658304Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6658580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6658702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6658983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6659120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6659125Z 
2022-11-23T01:33:43.6659198Z Running tests...
2022-11-23T01:33:43.6659375Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6659620Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18555
2022-11-23T01:33:43.6659777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18556
2022-11-23T01:33:43.6659930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18557
2022-11-23T01:33:43.6660202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6660324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6660610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6660746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6660924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6661223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6661346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6661625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6661760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6661936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6662208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6662360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6662637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6662774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6662940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6663239Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6663536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6663826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6663996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6664159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6664322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6664428Z skip: Need at least 3 CUDA devices (2.174s)
2022-11-23T01:33:43.6664433Z 
2022-11-23T01:33:43.6664630Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6664690Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6664695Z 
2022-11-23T01:33:43.6664763Z OK (skipped=1)
2022-11-23T01:33:43.6664768Z 
2022-11-23T01:33:43.6664853Z Generating XML reports...
2022-11-23T01:33:43.6665196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012830.xml
2022-11-23T01:33:43.6665425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6665702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6665825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6666102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6666226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6666247Z 
2022-11-23T01:33:43.6666307Z Running tests...
2022-11-23T01:33:43.6666500Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6666735Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18608
2022-11-23T01:33:43.6666891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18609
2022-11-23T01:33:43.6667045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18610
2022-11-23T01:33:43.6667319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6667443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6667751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6667873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6668051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6668323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6668445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6668725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6668887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6669063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6669336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6669446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6669723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6669857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6670032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6670328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6670631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6670918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6671084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6671243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6671392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6671567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6671740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6671911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6672202Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6672498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6672787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6672856Z ok (2.376s)
2022-11-23T01:33:43.6672861Z 
2022-11-23T01:33:43.6673055Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6673116Z Ran 1 test in 2.376s
2022-11-23T01:33:43.6673121Z 
2022-11-23T01:33:43.6673181Z OK
2022-11-23T01:33:43.6673185Z 
2022-11-23T01:33:43.6673269Z Generating XML reports...
2022-11-23T01:33:43.6673613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml
2022-11-23T01:33:43.6673848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6674121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6674245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6674552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6674686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6674691Z 
2022-11-23T01:33:43.6674749Z Running tests...
2022-11-23T01:33:43.6674944Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6675171Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18670
2022-11-23T01:33:43.6675328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18671
2022-11-23T01:33:43.6675512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18672
2022-11-23T01:33:43.6675786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6675910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6676193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6676316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6676491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6676762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6676885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6677168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6677300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6677475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6677748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6677869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6678134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6678269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6678443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6678742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6679039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6679333Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6679499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6679667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6679828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6679990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6680163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6680339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6680633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6680930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6681249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6681319Z ok (2.272s)
2022-11-23T01:33:43.6681324Z 
2022-11-23T01:33:43.6681519Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6681597Z Ran 1 test in 2.272s
2022-11-23T01:33:43.6681601Z 
2022-11-23T01:33:43.6681648Z OK
2022-11-23T01:33:43.6681653Z 
2022-11-23T01:33:43.6681736Z Generating XML reports...
2022-11-23T01:33:43.6682079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml
2022-11-23T01:33:43.6682342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6682616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6682740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6683022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6683159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6683164Z 
2022-11-23T01:33:43.6683236Z Running tests...
2022-11-23T01:33:43.6683418Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6684128Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2022-11-23T01:33:43.6684136Z 
2022-11-23T01:33:43.6684329Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6684405Z Ran 1 test in 0.257s
2022-11-23T01:33:43.6684410Z 
2022-11-23T01:33:43.6684481Z OK (skipped=1)
2022-11-23T01:33:43.6684485Z 
2022-11-23T01:33:43.6684569Z Generating XML reports...
2022-11-23T01:33:43.6684911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012842.xml
2022-11-23T01:33:43.6685144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6685420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6685532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6685814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6685949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6685953Z 
2022-11-23T01:33:43.6686027Z Running tests...
2022-11-23T01:33:43.6686223Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6686555Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2022-11-23T01:33:43.6686560Z 
2022-11-23T01:33:43.6686754Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6686829Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6686834Z 
2022-11-23T01:33:43.6686904Z OK (skipped=1)
2022-11-23T01:33:43.6686908Z 
2022-11-23T01:33:43.6686978Z Generating XML reports...
2022-11-23T01:33:43.6687320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml
2022-11-23T01:33:43.6687552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6687941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6688066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6688350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6688486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6688491Z 
2022-11-23T01:33:43.6688563Z Running tests...
2022-11-23T01:33:43.6688753Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6688986Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18749
2022-11-23T01:33:43.6689174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18750
2022-11-23T01:33:43.6689327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18751
2022-11-23T01:33:43.6689603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6689725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6690007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6690142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6690320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6690594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6690705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6690985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6691119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6691297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6691596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6691866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6691987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6692268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6692388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6692562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6692857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6693152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6693315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6693478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6693641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6693706Z ok (2.274s)
2022-11-23T01:33:43.6693711Z 
2022-11-23T01:33:43.6693908Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6693970Z Ran 1 test in 2.275s
2022-11-23T01:33:43.6693975Z 
2022-11-23T01:33:43.6694037Z OK
2022-11-23T01:33:43.6694041Z 
2022-11-23T01:33:43.6694123Z Generating XML reports...
2022-11-23T01:33:43.6694492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012846.xml
2022-11-23T01:33:43.6694727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6695134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6695261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6695543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6695666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6695745Z 
2022-11-23T01:33:43.6695805Z Running tests...
2022-11-23T01:33:43.6695998Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6696220Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18802
2022-11-23T01:33:43.6696380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18803
2022-11-23T01:33:43.6696533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18804
2022-11-23T01:33:43.6696805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6696927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6697207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6697328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6697506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6697779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6697901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6698182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6698316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6698494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6698763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6698884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6699150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6699282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6699457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6699757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6700055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6700345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6700507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6700665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6700918Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6701150Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6701347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6701596Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6702011Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6702017Z 
2022-11-23T01:33:43.6702609Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6702642Z 
2022-11-23T01:33:43.6702647Z 
2022-11-23T01:33:43.6702907Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6703149Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6703388Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6703635Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6703891Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6704132Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6704356Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6704605Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6704860Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6705106Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6705359Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6705597Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6705843Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6706096Z STAGE:2022-11-23 01:28:52 18802:18802 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6706339Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6706573Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6706828Z STAGE:2022-11-23 01:28:52 18803:18803 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6707077Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6707331Z STAGE:2022-11-23 01:28:52 18804:18804 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6707400Z ok (2.274s)
2022-11-23T01:33:43.6707404Z 
2022-11-23T01:33:43.6707599Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6707673Z Ran 1 test in 2.274s
2022-11-23T01:33:43.6707678Z 
2022-11-23T01:33:43.6707739Z OK
2022-11-23T01:33:43.6707743Z 
2022-11-23T01:33:43.6707814Z Generating XML reports...
2022-11-23T01:33:43.6708159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012850.xml
2022-11-23T01:33:43.6708392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6708668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6708821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6709107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6709242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6709247Z 
2022-11-23T01:33:43.6709319Z Running tests...
2022-11-23T01:33:43.6709513Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6709731Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18858
2022-11-23T01:33:43.6709918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18859
2022-11-23T01:33:43.6710070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18860
2022-11-23T01:33:43.6710347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6710470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6710752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6710887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6711064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6711337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6711449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6711728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6711862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6712040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6712310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6712433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6712711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6712842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6713005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6713304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6713605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6713899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6714063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6714228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6714388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6714457Z ok (2.073s)
2022-11-23T01:33:43.6714462Z 
2022-11-23T01:33:43.6714661Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6714723Z Ran 1 test in 2.074s
2022-11-23T01:33:43.6714728Z 
2022-11-23T01:33:43.6714789Z OK
2022-11-23T01:33:43.6714794Z 
2022-11-23T01:33:43.6714877Z Generating XML reports...
2022-11-23T01:33:43.6715243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012854.xml
2022-11-23T01:33:43.6715476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6715751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6715874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6716157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6716293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6716335Z 
2022-11-23T01:33:43.6716395Z Running tests...
2022-11-23T01:33:43.6716589Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6716773Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-11-23T01:33:43.6716779Z 
2022-11-23T01:33:43.6716971Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6717046Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6717051Z 
2022-11-23T01:33:43.6717121Z OK (skipped=1)
2022-11-23T01:33:43.6717126Z 
2022-11-23T01:33:43.6717208Z Generating XML reports...
2022-11-23T01:33:43.6717543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml
2022-11-23T01:33:43.6717761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6718039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6718165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6718444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6718576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6718581Z 
2022-11-23T01:33:43.6718655Z Running tests...
2022-11-23T01:33:43.6718847Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6719080Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18921
2022-11-23T01:33:43.6719236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18922
2022-11-23T01:33:43.6719376Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18923
2022-11-23T01:33:43.6719651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6719776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6720055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6720192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6720370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6720644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6720771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6721051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6721172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6721352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6721623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6721744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6722051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6722184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6722358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6722658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6722945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6723267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6723431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6723594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6723754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6723929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6724228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6724403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6724577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6724862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6725155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6725566Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6725572Z 
2022-11-23T01:33:43.6725823Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6726082Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6726325Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6726566Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6726813Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6727070Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6727317Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6727559Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6727877Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6728126Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6728383Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6728624Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6728860Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6729133Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6729390Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6729635Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6729876Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6730122Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6730365Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6730688Z STAGE:2022-11-23 01:29:01 18921:18921 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6730927Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6731173Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6731425Z STAGE:2022-11-23 01:29:01 18922:18922 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6731669Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6731924Z STAGE:2022-11-23 01:29:01 18923:18923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6731979Z ok (2.372s)
2022-11-23T01:33:43.6731984Z 
2022-11-23T01:33:43.6732181Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6732258Z Ran 1 test in 2.373s
2022-11-23T01:33:43.6732262Z 
2022-11-23T01:33:43.6732324Z OK
2022-11-23T01:33:43.6732328Z 
2022-11-23T01:33:43.6732414Z Generating XML reports...
2022-11-23T01:33:43.6732757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012859.xml
2022-11-23T01:33:43.6732990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6733268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6733393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6733662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6733799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6733804Z 
2022-11-23T01:33:43.6733879Z Running tests...
2022-11-23T01:33:43.6734074Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6734301Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18986
2022-11-23T01:33:43.6734458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18987
2022-11-23T01:33:43.6734615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18988
2022-11-23T01:33:43.6735002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6735119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6735403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6735537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6735718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6735989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6736111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6736441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6736575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6736752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6737011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6737133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6737410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6737576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6737751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6738054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6738350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6738641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6738805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6738958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6739122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6739297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6739470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6739645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6739940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6740234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6740480Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6740775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6741009Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6741257Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6741516Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6741762Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6742020Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6742260Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6742498Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6742744Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6742998Z STAGE:2022-11-23 01:29:05 18987:18987 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6743230Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6743510Z STAGE:2022-11-23 01:29:05 18988:18988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6743579Z ok (2.276s)
2022-11-23T01:33:43.6743584Z 
2022-11-23T01:33:43.6743779Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6743855Z Ran 1 test in 2.276s
2022-11-23T01:33:43.6743860Z 
2022-11-23T01:33:43.6743921Z OK
2022-11-23T01:33:43.6743925Z 
2022-11-23T01:33:43.6744009Z Generating XML reports...
2022-11-23T01:33:43.6744350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012903.xml
2022-11-23T01:33:43.6744614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6744890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6745016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6745298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6745434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6745439Z 
2022-11-23T01:33:43.6745511Z Running tests...
2022-11-23T01:33:43.6745705Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6745935Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19047
2022-11-23T01:33:43.6746090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19048
2022-11-23T01:33:43.6746234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19049
2022-11-23T01:33:43.6746508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6746630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6746914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6747049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6747227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6747499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6747623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6747907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6748044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6748221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6748523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6748793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6748916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6749193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6749327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6749502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6749790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6750085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6750278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6750443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6750602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6750671Z ok (2.173s)
2022-11-23T01:33:43.6750675Z 
2022-11-23T01:33:43.6750870Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6750945Z Ran 1 test in 2.174s
2022-11-23T01:33:43.6750950Z 
2022-11-23T01:33:43.6751039Z OK
2022-11-23T01:33:43.6751045Z 
2022-11-23T01:33:43.6751115Z Generating XML reports...
2022-11-23T01:33:43.6751460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012907.xml
2022-11-23T01:33:43.6751692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6751968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6752092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6752373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6752507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6752511Z 
2022-11-23T01:33:43.6752582Z Running tests...
2022-11-23T01:33:43.6752764Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6753454Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2022-11-23T01:33:43.6753475Z 
2022-11-23T01:33:43.6753656Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6753731Z Ran 1 test in 0.256s
2022-11-23T01:33:43.6753736Z 
2022-11-23T01:33:43.6753807Z OK (skipped=1)
2022-11-23T01:33:43.6753812Z 
2022-11-23T01:33:43.6753896Z Generating XML reports...
2022-11-23T01:33:43.6754234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012911.xml
2022-11-23T01:33:43.6754460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6754733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6754856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6755120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6755255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6755259Z 
2022-11-23T01:33:43.6755332Z Running tests...
2022-11-23T01:33:43.6755523Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6755748Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19110
2022-11-23T01:33:43.6755902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19111
2022-11-23T01:33:43.6756054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19112
2022-11-23T01:33:43.6756331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6756452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6756721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6756883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6757061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6757329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6757451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6757810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6757977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6758153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6758422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6758533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6758812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6758948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6759121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6759421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6759725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6760018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6760182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6760348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6760497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6760673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6760847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6761017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6761311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6761604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6761895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6761963Z ok (2.374s)
2022-11-23T01:33:43.6761968Z 
2022-11-23T01:33:43.6762162Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6762224Z Ran 1 test in 2.374s
2022-11-23T01:33:43.6762229Z 
2022-11-23T01:33:43.6762288Z OK
2022-11-23T01:33:43.6762292Z 
2022-11-23T01:33:43.6762377Z Generating XML reports...
2022-11-23T01:33:43.6762716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012913.xml
2022-11-23T01:33:43.6762949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6763222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6763345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6763652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6763776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6763796Z 
2022-11-23T01:33:43.6763853Z Running tests...
2022-11-23T01:33:43.6764049Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6764275Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19169
2022-11-23T01:33:43.6764434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19170
2022-11-23T01:33:43.6764618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19171
2022-11-23T01:33:43.6764943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6765069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6765356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6765478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6765656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6765927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6766050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6766329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6766466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6766644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6766942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6767198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6767318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6767597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6767798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6767973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6768273Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6768568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6768738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6768902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6769051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6769118Z ok (2.073s)
2022-11-23T01:33:43.6769124Z 
2022-11-23T01:33:43.6769319Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6769394Z Ran 1 test in 2.073s
2022-11-23T01:33:43.6769399Z 
2022-11-23T01:33:43.6769463Z OK
2022-11-23T01:33:43.6769470Z 
2022-11-23T01:33:43.6769553Z Generating XML reports...
2022-11-23T01:33:43.6769894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012917.xml
2022-11-23T01:33:43.6770126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6770435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6770547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6770829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6770962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6770967Z 
2022-11-23T01:33:43.6771039Z Running tests...
2022-11-23T01:33:43.6771234Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6771482Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19222
2022-11-23T01:33:43.6771637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19223
2022-11-23T01:33:43.6771791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19224
2022-11-23T01:33:43.6772052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6772177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6772460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6772592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6772770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6773045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6773166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6773448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6773585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6773750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6774019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6774140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6774418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6774550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6774728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6775133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6775437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6775728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6775880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6776038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6776202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6776272Z ok (2.374s)
2022-11-23T01:33:43.6776278Z 
2022-11-23T01:33:43.6776479Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6776554Z Ran 1 test in 2.374s
2022-11-23T01:33:43.6776559Z 
2022-11-23T01:33:43.6776620Z OK
2022-11-23T01:33:43.6776624Z 
2022-11-23T01:33:43.6776708Z Generating XML reports...
2022-11-23T01:33:43.6777093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012921.xml
2022-11-23T01:33:43.6777327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6777603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6777728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6778008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6778177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6778182Z 
2022-11-23T01:33:43.6778256Z Running tests...
2022-11-23T01:33:43.6778451Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6778692Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19275
2022-11-23T01:33:43.6778836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19276
2022-11-23T01:33:43.6778989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19277
2022-11-23T01:33:43.6779264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6779388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6779665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6779802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6779979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6780252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6780364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6780642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6780775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6780952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6781222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6781348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6781626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6781759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6781937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6782223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6782522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6782816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6782979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6783143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6783303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6783476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6783680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6783973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6784131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6784429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6784721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6784817Z ok (2.275s)
2022-11-23T01:33:43.6784822Z 
2022-11-23T01:33:43.6785018Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6785093Z Ran 1 test in 2.275s
2022-11-23T01:33:43.6785098Z 
2022-11-23T01:33:43.6785157Z OK
2022-11-23T01:33:43.6785161Z 
2022-11-23T01:33:43.6785245Z Generating XML reports...
2022-11-23T01:33:43.6785587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012925.xml
2022-11-23T01:33:43.6785807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6786083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6786207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6786488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6786625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6786630Z 
2022-11-23T01:33:43.6786704Z Running tests...
2022-11-23T01:33:43.6786897Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6787132Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19337
2022-11-23T01:33:43.6787275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19338
2022-11-23T01:33:43.6787429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19339
2022-11-23T01:33:43.6787704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6787827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6788109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6788246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6788423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6788698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6788821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6789090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6789225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6789401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6789668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6789792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6790073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6790206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6790411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6790710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6790992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6791292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6791495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6791659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6791832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6792008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6792168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6792342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6792635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6792914Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6793206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6793273Z ok (2.275s)
2022-11-23T01:33:43.6793278Z 
2022-11-23T01:33:43.6793477Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6793552Z Ran 1 test in 2.275s
2022-11-23T01:33:43.6793557Z 
2022-11-23T01:33:43.6793618Z OK
2022-11-23T01:33:43.6793623Z 
2022-11-23T01:33:43.6793706Z Generating XML reports...
2022-11-23T01:33:43.6794050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012929.xml
2022-11-23T01:33:43.6794267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6794543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6794665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6794949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6795083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6795087Z 
2022-11-23T01:33:43.6795158Z Running tests...
2022-11-23T01:33:43.6795354Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6795674Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.002s)
2022-11-23T01:33:43.6795679Z 
2022-11-23T01:33:43.6795870Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6795930Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6795949Z 
2022-11-23T01:33:43.6796007Z OK (skipped=1)
2022-11-23T01:33:43.6796011Z 
2022-11-23T01:33:43.6796093Z Generating XML reports...
2022-11-23T01:33:43.6796435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012933.xml
2022-11-23T01:33:43.6796668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6796944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6797093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6797377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6797511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6797516Z 
2022-11-23T01:33:43.6797574Z Running tests...
2022-11-23T01:33:43.6797768Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6797990Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19406
2022-11-23T01:33:43.6798175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19407
2022-11-23T01:33:43.6798328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19408
2022-11-23T01:33:43.6798600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6798724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6799011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6799144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6799308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6799582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6799708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6799988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6800121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6800300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6800571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6800693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6800959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6801093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6801269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6801570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6801868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6802161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6802323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6802486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6802647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6802702Z ok (2.275s)
2022-11-23T01:33:43.6802707Z 
2022-11-23T01:33:43.6802903Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6802979Z Ran 1 test in 2.275s
2022-11-23T01:33:43.6802984Z 
2022-11-23T01:33:43.6803044Z OK
2022-11-23T01:33:43.6803048Z 
2022-11-23T01:33:43.6803131Z Generating XML reports...
2022-11-23T01:33:43.6803473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012935.xml
2022-11-23T01:33:43.6803732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6804010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6804132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6804398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6804534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6804538Z 
2022-11-23T01:33:43.6804612Z Running tests...
2022-11-23T01:33:43.6804834Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6805054Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19459
2022-11-23T01:33:43.6805211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19460
2022-11-23T01:33:43.6805368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19461
2022-11-23T01:33:43.6805643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6805751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6806031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6806166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6806342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6806616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6806737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6807017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6807150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6807329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6807585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6807788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6808069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6808203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6808377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6808678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6808972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6809263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6809426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6809570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6809733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6809801Z ok (2.274s)
2022-11-23T01:33:43.6809807Z 
2022-11-23T01:33:43.6810007Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6810080Z Ran 1 test in 2.274s
2022-11-23T01:33:43.6810085Z 
2022-11-23T01:33:43.6810145Z OK
2022-11-23T01:33:43.6810149Z 
2022-11-23T01:33:43.6810262Z Generating XML reports...
2022-11-23T01:33:43.6810602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012939.xml
2022-11-23T01:33:43.6810820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6811096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6811218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6811501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6811668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6811673Z 
2022-11-23T01:33:43.6811745Z Running tests...
2022-11-23T01:33:43.6811940Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6812181Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19512
2022-11-23T01:33:43.6812339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19513
2022-11-23T01:33:43.6812480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19514
2022-11-23T01:33:43.6812752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6812875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6813156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6813289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6813463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6813737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6813860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6814125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6814259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6814432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6814704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6814827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6815271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6815408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6815582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6815880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6816165Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6816459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6816622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6816786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6817035Z STAGE:2022-11-23 01:29:45 19513:19513 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6817340Z STAGE:2022-11-23 01:29:45 19514:19514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6817504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6817744Z STAGE:2022-11-23 01:29:45 19512:19512 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6817990Z STAGE:2022-11-23 01:29:45 19512:19512 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6818235Z STAGE:2022-11-23 01:29:45 19512:19512 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6818522Z STAGE:2022-11-23 01:29:45 19513:19513 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6818780Z STAGE:2022-11-23 01:29:45 19513:19513 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6819026Z STAGE:2022-11-23 01:29:45 19514:19514 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6819283Z STAGE:2022-11-23 01:29:45 19514:19514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6819352Z ok (2.271s)
2022-11-23T01:33:43.6819357Z 
2022-11-23T01:33:43.6819556Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6819629Z Ran 1 test in 2.272s
2022-11-23T01:33:43.6819634Z 
2022-11-23T01:33:43.6819694Z OK
2022-11-23T01:33:43.6819698Z 
2022-11-23T01:33:43.6819767Z Generating XML reports...
2022-11-23T01:33:43.6820109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012943.xml
2022-11-23T01:33:43.6820346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6820625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6820747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6821032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6821169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6821174Z 
2022-11-23T01:33:43.6821247Z Running tests...
2022-11-23T01:33:43.6821428Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6821664Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19568
2022-11-23T01:33:43.6821822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19569
2022-11-23T01:33:43.6821977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19570
2022-11-23T01:33:43.6822251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6822372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6822656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6822794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6822974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6823232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6823354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6823633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6823765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6823942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6824269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6824540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6824665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6824950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6825071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6825274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6825571Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6825870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6826035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6826199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6826452Z STAGE:2022-11-23 01:29:49 19568:19568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6826614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6826847Z STAGE:2022-11-23 01:29:49 19570:19570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6827091Z STAGE:2022-11-23 01:29:49 19569:19569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6827338Z STAGE:2022-11-23 01:29:49 19568:19568 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6827593Z STAGE:2022-11-23 01:29:49 19568:19568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6827842Z STAGE:2022-11-23 01:29:49 19570:19570 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6828099Z STAGE:2022-11-23 01:29:49 19570:19570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6828346Z STAGE:2022-11-23 01:29:49 19569:19569 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6828599Z STAGE:2022-11-23 01:29:49 19569:19569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6828665Z ok (2.177s)
2022-11-23T01:33:43.6828672Z 
2022-11-23T01:33:43.6828855Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6828930Z Ran 1 test in 2.177s
2022-11-23T01:33:43.6828936Z 
2022-11-23T01:33:43.6828993Z OK
2022-11-23T01:33:43.6828998Z 
2022-11-23T01:33:43.6829082Z Generating XML reports...
2022-11-23T01:33:43.6829461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012947.xml
2022-11-23T01:33:43.6829694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6829970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6830093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6830375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6830497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6830504Z 
2022-11-23T01:33:43.6830577Z Running tests...
2022-11-23T01:33:43.6830773Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6831115Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2022-11-23T01:33:43.6831152Z 
2022-11-23T01:33:43.6831344Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6831418Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6831423Z 
2022-11-23T01:33:43.6831495Z OK (skipped=1)
2022-11-23T01:33:43.6831499Z 
2022-11-23T01:33:43.6831582Z Generating XML reports...
2022-11-23T01:33:43.6831925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012951.xml
2022-11-23T01:33:43.6832145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6832447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6832571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6832852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6832990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6832995Z 
2022-11-23T01:33:43.6833071Z Running tests...
2022-11-23T01:33:43.6833262Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6833628Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl', 'ucc'} (0.001s)
2022-11-23T01:33:43.6833634Z 
2022-11-23T01:33:43.6833829Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6833890Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6833896Z 
2022-11-23T01:33:43.6833968Z OK (skipped=1)
2022-11-23T01:33:43.6833972Z 
2022-11-23T01:33:43.6834055Z Generating XML reports...
2022-11-23T01:33:43.6834395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012952.xml
2022-11-23T01:33:43.6834627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6834902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6835024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6835306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6835450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6835468Z 
2022-11-23T01:33:43.6835525Z Running tests...
2022-11-23T01:33:43.6835720Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6835966Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19644
2022-11-23T01:33:43.6836128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19645
2022-11-23T01:33:43.6836283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19646
2022-11-23T01:33:43.6836555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6836663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6836944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6837077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6837252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6837524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6837646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6837985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6838119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6838283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6838552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6838673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6838951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6839133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6839357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6839659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6839957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6840249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6840412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6840562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6840725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6840983Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:33:43.6841149Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:33:43.6841217Z ok (4.276s)
2022-11-23T01:33:43.6841222Z 
2022-11-23T01:33:43.6841417Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6841489Z Ran 1 test in 4.276s
2022-11-23T01:33:43.6841494Z 
2022-11-23T01:33:43.6841551Z OK
2022-11-23T01:33:43.6841556Z 
2022-11-23T01:33:43.6841626Z Generating XML reports...
2022-11-23T01:33:43.6841968Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012954.xml
2022-11-23T01:33:43.6842199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6842473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6842596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6842874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6843010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6843015Z 
2022-11-23T01:33:43.6843086Z Running tests...
2022-11-23T01:33:43.6843277Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6843501Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19697
2022-11-23T01:33:43.6843658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19698
2022-11-23T01:33:43.6843812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19699
2022-11-23T01:33:43.6844084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6844202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6844508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6844644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6844817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6845076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6845197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6845472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6845633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6845808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6846080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6846200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6846476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6846607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6846769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6847067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6847365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6847657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6847899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6848062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6848222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6848383Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:33:43.6848642Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:33:43.6848699Z ok (4.375s)
2022-11-23T01:33:43.6848717Z 
2022-11-23T01:33:43.6848898Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6848970Z Ran 1 test in 4.375s
2022-11-23T01:33:43.6848975Z 
2022-11-23T01:33:43.6849033Z OK
2022-11-23T01:33:43.6849038Z 
2022-11-23T01:33:43.6849120Z Generating XML reports...
2022-11-23T01:33:43.6849464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013000.xml
2022-11-23T01:33:43.6849695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6849968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6850090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6850359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6850494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6850499Z 
2022-11-23T01:33:43.6850570Z Running tests...
2022-11-23T01:33:43.6850765Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6851046Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19750
2022-11-23T01:33:43.6851202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19751
2022-11-23T01:33:43.6851354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19752
2022-11-23T01:33:43.6851625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6851733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6852009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6852170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6852344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6852617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6852741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6853021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6853153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6853331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6853586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6853708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6853984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6854116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6854290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6854584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6854972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6855270Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6855431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6855584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6855745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6855920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6856092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6856263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6856554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6856850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6857141Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6857298Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T01:33:43.6857408Z No ranks successfully processed in monitoredBarrier.
2022-11-23T01:33:43.6857570Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T01:33:43.6857681Z ok (2.273s)
2022-11-23T01:33:43.6857687Z 
2022-11-23T01:33:43.6857884Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6857956Z Ran 1 test in 2.273s
2022-11-23T01:33:43.6857961Z 
2022-11-23T01:33:43.6858020Z OK
2022-11-23T01:33:43.6858025Z 
2022-11-23T01:33:43.6858106Z Generating XML reports...
2022-11-23T01:33:43.6858444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013006.xml
2022-11-23T01:33:43.6858661Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6858966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6859088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6859368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6859504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6859509Z 
2022-11-23T01:33:43.6859579Z Running tests...
2022-11-23T01:33:43.6859772Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6860019Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19812
2022-11-23T01:33:43.6860174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19813
2022-11-23T01:33:43.6860313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19814
2022-11-23T01:33:43.6860589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6860712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6860992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6861124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6861298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6861569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6861689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6861956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6862090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6862263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6862531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6862654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6862932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6863061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6863232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6863529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6863816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6864108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6864299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6864464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6864624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6864798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6864969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6865138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6865508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6865829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6866120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6866286Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:33:43.6866658Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:444: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-11-23T01:33:43.6866787Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-11-23T01:33:43.6866855Z ok (2.477s)
2022-11-23T01:33:43.6866862Z 
2022-11-23T01:33:43.6867059Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6867131Z Ran 1 test in 2.477s
2022-11-23T01:33:43.6867136Z 
2022-11-23T01:33:43.6867194Z OK
2022-11-23T01:33:43.6867198Z 
2022-11-23T01:33:43.6867273Z Generating XML reports...
2022-11-23T01:33:43.6867615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013010.xml
2022-11-23T01:33:43.6867847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6868121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6868244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6868521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6868654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6868662Z 
2022-11-23T01:33:43.6868732Z Running tests...
2022-11-23T01:33:43.6868926Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6869158Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19871
2022-11-23T01:33:43.6869315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19872
2022-11-23T01:33:43.6869466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19873
2022-11-23T01:33:43.6869739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6869859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6870135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6870269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6870443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6870702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6870856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6871138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6871270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6871446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6871714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6871836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6872140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6872272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6872433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6872731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6873025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6873316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6873480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6873642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6873804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6873967Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:33:43.6874128Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:33:43.6874276Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:33:43.6874342Z ok (2.374s)
2022-11-23T01:33:43.6874346Z 
2022-11-23T01:33:43.6874542Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6874613Z Ran 1 test in 2.375s
2022-11-23T01:33:43.6874619Z 
2022-11-23T01:33:43.6874677Z OK
2022-11-23T01:33:43.6874681Z 
2022-11-23T01:33:43.6874762Z Generating XML reports...
2022-11-23T01:33:43.6875102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013014.xml
2022-11-23T01:33:43.6875335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6875596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6875718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6875998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6876132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6876137Z 
2022-11-23T01:33:43.6876208Z Running tests...
2022-11-23T01:33:43.6876398Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6876697Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:33:43.6876704Z 
2022-11-23T01:33:43.6876890Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6876963Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6876968Z 
2022-11-23T01:33:43.6877025Z OK (skipped=1)
2022-11-23T01:33:43.6877040Z 
2022-11-23T01:33:43.6877110Z Generating XML reports...
2022-11-23T01:33:43.6877482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013019.xml
2022-11-23T01:33:43.6877716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6877987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6878107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6878386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6878519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6878553Z 
2022-11-23T01:33:43.6878625Z Running tests...
2022-11-23T01:33:43.6878805Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6879105Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:33:43.6879110Z 
2022-11-23T01:33:43.6879301Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6879374Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6879379Z 
2022-11-23T01:33:43.6879451Z OK (skipped=1)
2022-11-23T01:33:43.6879455Z 
2022-11-23T01:33:43.6879537Z Generating XML reports...
2022-11-23T01:33:43.6879872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml
2022-11-23T01:33:43.6880106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6880380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6880489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6880769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6880904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6880909Z 
2022-11-23T01:33:43.6880981Z Running tests...
2022-11-23T01:33:43.6881172Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6881473Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:33:43.6881478Z 
2022-11-23T01:33:43.6881666Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6881737Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6881742Z 
2022-11-23T01:33:43.6881801Z OK (skipped=1)
2022-11-23T01:33:43.6881819Z 
2022-11-23T01:33:43.6881888Z Generating XML reports...
2022-11-23T01:33:43.6882226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013022.xml
2022-11-23T01:33:43.6882457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6882730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6882851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6883131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6883263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6883268Z 
2022-11-23T01:33:43.6883338Z Running tests...
2022-11-23T01:33:43.6883518Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6883818Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:33:43.6883822Z 
2022-11-23T01:33:43.6884012Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6884084Z Ran 1 test in 0.003s
2022-11-23T01:33:43.6884089Z 
2022-11-23T01:33:43.6884183Z OK (skipped=1)
2022-11-23T01:33:43.6884188Z 
2022-11-23T01:33:43.6884274Z Generating XML reports...
2022-11-23T01:33:43.6884613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013023.xml
2022-11-23T01:33:43.6884842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6885112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6885221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6885541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6885674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6885678Z 
2022-11-23T01:33:43.6885748Z Running tests...
2022-11-23T01:33:43.6885941Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6886155Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T01:33:43.6886160Z 
2022-11-23T01:33:43.6886350Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6886423Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6886427Z 
2022-11-23T01:33:43.6886497Z OK (skipped=1)
2022-11-23T01:33:43.6886502Z 
2022-11-23T01:33:43.6886572Z Generating XML reports...
2022-11-23T01:33:43.6886907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013025.xml
2022-11-23T01:33:43.6887138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6887409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6887532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6887887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6888024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6888029Z 
2022-11-23T01:33:43.6888099Z Running tests...
2022-11-23T01:33:43.6888277Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6888456Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:33:43.6888463Z 
2022-11-23T01:33:43.6888653Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6888725Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6888730Z 
2022-11-23T01:33:43.6888799Z OK (skipped=1)
2022-11-23T01:33:43.6888804Z 
2022-11-23T01:33:43.6888885Z Generating XML reports...
2022-11-23T01:33:43.6889220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013027.xml
2022-11-23T01:33:43.6889450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6889722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6889831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6890107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6890240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6890246Z 
2022-11-23T01:33:43.6890317Z Running tests...
2022-11-23T01:33:43.6890505Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6890701Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:33:43.6890705Z 
2022-11-23T01:33:43.6890924Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6890997Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6891002Z 
2022-11-23T01:33:43.6891071Z OK (skipped=1)
2022-11-23T01:33:43.6891075Z 
2022-11-23T01:33:43.6891145Z Generating XML reports...
2022-11-23T01:33:43.6891478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013028.xml
2022-11-23T01:33:43.6891707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6892008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6892126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6892404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6892539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6892544Z 
2022-11-23T01:33:43.6892614Z Running tests...
2022-11-23T01:33:43.6892790Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6893016Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:33:43.6893021Z 
2022-11-23T01:33:43.6893212Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6893284Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6893291Z 
2022-11-23T01:33:43.6893364Z OK (skipped=1)
2022-11-23T01:33:43.6893368Z 
2022-11-23T01:33:43.6893448Z Generating XML reports...
2022-11-23T01:33:43.6893774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013030.xml
2022-11-23T01:33:43.6894002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6894274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6894395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6894658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6894788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6894793Z 
2022-11-23T01:33:43.6894961Z Running tests...
2022-11-23T01:33:43.6895152Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6895417Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20004
2022-11-23T01:33:43.6895572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20005
2022-11-23T01:33:43.6895728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20006
2022-11-23T01:33:43.6895996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6896104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6896384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6896519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6896696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6896967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6897087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6897414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6897546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6897720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6897975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6898094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6898369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6898537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6898712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6899010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6899306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6899600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6899763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6899914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6900076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6900173Z skip: CUDA is not available. (2.174s)
2022-11-23T01:33:43.6900178Z 
2022-11-23T01:33:43.6900370Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6900442Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6900447Z 
2022-11-23T01:33:43.6900515Z OK (skipped=1)
2022-11-23T01:33:43.6900519Z 
2022-11-23T01:33:43.6900603Z Generating XML reports...
2022-11-23T01:33:43.6900946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013031.xml
2022-11-23T01:33:43.6901164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6901436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6901556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6901833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6901967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6901972Z 
2022-11-23T01:33:43.6902041Z Running tests...
2022-11-23T01:33:43.6902236Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6902492Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20057
2022-11-23T01:33:43.6902647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20058
2022-11-23T01:33:43.6902787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20059
2022-11-23T01:33:43.6903058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6903178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6903461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6903594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6903769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6904066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6904187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6904465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6904586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6904763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6905062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6905181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6905456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6905591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6905763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6906061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6906347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6906637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6906802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6906967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6907126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6907219Z skip: CUDA is not available. (2.176s)
2022-11-23T01:33:43.6907225Z 
2022-11-23T01:33:43.6907418Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6907490Z Ran 1 test in 2.177s
2022-11-23T01:33:43.6907495Z 
2022-11-23T01:33:43.6907563Z OK (skipped=1)
2022-11-23T01:33:43.6907568Z 
2022-11-23T01:33:43.6907637Z Generating XML reports...
2022-11-23T01:33:43.6907976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013035.xml
2022-11-23T01:33:43.6908206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6908479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6908599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6908883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6909016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6909020Z 
2022-11-23T01:33:43.6909089Z Running tests...
2022-11-23T01:33:43.6909280Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6909467Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:33:43.6909472Z 
2022-11-23T01:33:43.6909661Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6909735Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6909740Z 
2022-11-23T01:33:43.6909807Z OK (skipped=1)
2022-11-23T01:33:43.6909812Z 
2022-11-23T01:33:43.6909891Z Generating XML reports...
2022-11-23T01:33:43.6910229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013039.xml
2022-11-23T01:33:43.6910485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6910756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6910876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6911145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6911279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6911284Z 
2022-11-23T01:33:43.6911353Z Running tests...
2022-11-23T01:33:43.6911576Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6911795Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.001s)
2022-11-23T01:33:43.6911800Z 
2022-11-23T01:33:43.6911986Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6912059Z Ran 1 test in 0.002s
2022-11-23T01:33:43.6912063Z 
2022-11-23T01:33:43.6912132Z OK (skipped=1)
2022-11-23T01:33:43.6912137Z 
2022-11-23T01:33:43.6912218Z Generating XML reports...
2022-11-23T01:33:43.6912542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013041.xml
2022-11-23T01:33:43.6912772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6913043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6913166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6913445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6913577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6913582Z 
2022-11-23T01:33:43.6913655Z Running tests...
2022-11-23T01:33:43.6913847Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6914550Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.253s)
2022-11-23T01:33:43.6914556Z 
2022-11-23T01:33:43.6914743Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6914807Z Ran 1 test in 0.253s
2022-11-23T01:33:43.6914811Z 
2022-11-23T01:33:43.6914880Z OK (skipped=1)
2022-11-23T01:33:43.6914884Z 
2022-11-23T01:33:43.6914965Z Generating XML reports...
2022-11-23T01:33:43.6915305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013042.xml
2022-11-23T01:33:43.6915535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6915808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6915930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6916208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6916328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6916347Z 
2022-11-23T01:33:43.6916405Z Running tests...
2022-11-23T01:33:43.6916598Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6916844Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20140
2022-11-23T01:33:43.6917026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20141
2022-11-23T01:33:43.6917180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20142
2022-11-23T01:33:43.6917455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6917578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6917854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6917974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6918180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6918451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6918573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6918857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6918988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6919165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6919432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6919540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6919821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6919952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6920125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6920423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6920721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6921008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6921170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6921332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6921482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6921585Z skip: Need at least 2 CUDA devices (2.175s)
2022-11-23T01:33:43.6921591Z 
2022-11-23T01:33:43.6921785Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6921860Z Ran 1 test in 2.175s
2022-11-23T01:33:43.6921865Z 
2022-11-23T01:33:43.6921935Z OK (skipped=1)
2022-11-23T01:33:43.6921939Z 
2022-11-23T01:33:43.6922022Z Generating XML reports...
2022-11-23T01:33:43.6922361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml
2022-11-23T01:33:43.6922588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6922861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6922974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6923252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6923384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6923389Z 
2022-11-23T01:33:43.6923459Z Running tests...
2022-11-23T01:33:43.6923690Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6923929Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20193
2022-11-23T01:33:43.6924083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20194
2022-11-23T01:33:43.6924233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20195
2022-11-23T01:33:43.6924504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6924639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6924919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6925051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6925230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6925500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6925621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6925898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6926029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6926193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6926462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6926580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6926859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6926990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6927164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6927462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6927836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6928135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6928288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6928443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6928604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6928704Z skip: Need at least 2 CUDA devices (2.074s)
2022-11-23T01:33:43.6928709Z 
2022-11-23T01:33:43.6928902Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6928975Z Ran 1 test in 2.074s
2022-11-23T01:33:43.6928980Z 
2022-11-23T01:33:43.6929048Z OK (skipped=1)
2022-11-23T01:33:43.6929053Z 
2022-11-23T01:33:43.6929136Z Generating XML reports...
2022-11-23T01:33:43.6929475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013048.xml
2022-11-23T01:33:43.6929693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6929968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6930088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6930399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6930533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6930537Z 
2022-11-23T01:33:43.6930609Z Running tests...
2022-11-23T01:33:43.6930802Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6931055Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20246
2022-11-23T01:33:43.6931244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20247
2022-11-23T01:33:43.6931397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20248
2022-11-23T01:33:43.6931672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6931796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6932076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6932208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6932385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6932656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6932777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6933045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6933176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6933352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6933621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6933743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6934019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6934151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6934324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6934626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6935026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6935327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6935491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6935655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6935816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6935916Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T01:33:43.6935921Z 
2022-11-23T01:33:43.6936116Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6936192Z Ran 1 test in 2.177s
2022-11-23T01:33:43.6936197Z 
2022-11-23T01:33:43.6936254Z OK (skipped=1)
2022-11-23T01:33:43.6936270Z 
2022-11-23T01:33:43.6936340Z Generating XML reports...
2022-11-23T01:33:43.6936681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml
2022-11-23T01:33:43.6936959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6937235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6937356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6937633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6937764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6937842Z 
2022-11-23T01:33:43.6937914Z Running tests...
2022-11-23T01:33:43.6938096Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6938799Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2022-11-23T01:33:43.6938816Z 
2022-11-23T01:33:43.6938995Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6939068Z Ran 1 test in 0.257s
2022-11-23T01:33:43.6939073Z 
2022-11-23T01:33:43.6939142Z OK (skipped=1)
2022-11-23T01:33:43.6939146Z 
2022-11-23T01:33:43.6939229Z Generating XML reports...
2022-11-23T01:33:43.6939566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013056.xml
2022-11-23T01:33:43.6939797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6940069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6940190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6940459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6940592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6940596Z 
2022-11-23T01:33:43.6940666Z Running tests...
2022-11-23T01:33:43.6940858Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6941574Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T01:33:43.6941582Z 
2022-11-23T01:33:43.6941772Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6941844Z Ran 1 test in 0.252s
2022-11-23T01:33:43.6941851Z 
2022-11-23T01:33:43.6941921Z OK (skipped=1)
2022-11-23T01:33:43.6941926Z 
2022-11-23T01:33:43.6942007Z Generating XML reports...
2022-11-23T01:33:43.6942346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml
2022-11-23T01:33:43.6942563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6942834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6942957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6943239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6943373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6943378Z 
2022-11-23T01:33:43.6943447Z Running tests...
2022-11-23T01:33:43.6943667Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6943936Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20319
2022-11-23T01:33:43.6944090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20320
2022-11-23T01:33:43.6944233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20321
2022-11-23T01:33:43.6944506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6944659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6944940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6945070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6945247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6945519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6945640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6945908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6946041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6946217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6946487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6946608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6946887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6947016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6947189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6947487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6947772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6948065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6948229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6948394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6948556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6948659Z skip: Need at least 4 CUDA devices (2.171s)
2022-11-23T01:33:43.6948664Z 
2022-11-23T01:33:43.6948859Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6948931Z Ran 1 test in 2.171s
2022-11-23T01:33:43.6948936Z 
2022-11-23T01:33:43.6949007Z OK (skipped=1)
2022-11-23T01:33:43.6949011Z 
2022-11-23T01:33:43.6949082Z Generating XML reports...
2022-11-23T01:33:43.6949419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml
2022-11-23T01:33:43.6949654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6949929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6950051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6950358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6950493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6950498Z 
2022-11-23T01:33:43.6950568Z Running tests...
2022-11-23T01:33:43.6950748Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6951026Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20372
2022-11-23T01:33:43.6951208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20373
2022-11-23T01:33:43.6951361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20374
2022-11-23T01:33:43.6951633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6951757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6952036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6952168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6952345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6952604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6952727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6953006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6953139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6953318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6953585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6953706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6953983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6954114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6954278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6954576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6954873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6955163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6955328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6955491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6955650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6955752Z skip: Need at least 4 CUDA devices (2.173s)
2022-11-23T01:33:43.6955757Z 
2022-11-23T01:33:43.6955951Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6956014Z Ran 1 test in 2.173s
2022-11-23T01:33:43.6956019Z 
2022-11-23T01:33:43.6956088Z OK (skipped=1)
2022-11-23T01:33:43.6956092Z 
2022-11-23T01:33:43.6956175Z Generating XML reports...
2022-11-23T01:33:43.6956515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013104.xml
2022-11-23T01:33:43.6956775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6957049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6957169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6957445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6957566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6957609Z 
2022-11-23T01:33:43.6957669Z Running tests...
2022-11-23T01:33:43.6957861Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6958576Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.257s)
2022-11-23T01:33:43.6958582Z 
2022-11-23T01:33:43.6958772Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6958847Z Ran 1 test in 0.258s
2022-11-23T01:33:43.6958851Z 
2022-11-23T01:33:43.6958921Z OK (skipped=1)
2022-11-23T01:33:43.6958926Z 
2022-11-23T01:33:43.6959008Z Generating XML reports...
2022-11-23T01:33:43.6959346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013108.xml
2022-11-23T01:33:43.6959579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6959841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6959962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6960246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6960383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6960387Z 
2022-11-23T01:33:43.6960459Z Running tests...
2022-11-23T01:33:43.6960650Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6960885Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20435
2022-11-23T01:33:43.6961040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20436
2022-11-23T01:33:43.6961191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20437
2022-11-23T01:33:43.6961473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6961595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6961872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6962009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6962185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6962453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6962574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6962853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6962986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6963150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6963454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6963575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6963852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6963985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6964159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6964484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6964777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6965068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6965220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6965382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6965543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6965717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6965890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6966063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6966451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6966750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6967037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6967274Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6967513Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6967816Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6968075Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6968319Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6968576Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6968834Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6969078Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6969332Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6969559Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6969798Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6970035Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6970277Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6970567Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6970814Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6971065Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6971306Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6971556Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6971783Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6972048Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6972284Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6972530Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6972781Z STAGE:2022-11-23 01:31:11 20435:20435 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6973024Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6973262Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6973516Z STAGE:2022-11-23 01:31:11 20436:20436 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6973770Z STAGE:2022-11-23 01:31:11 20437:20437 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6973825Z ok (2.274s)
2022-11-23T01:33:43.6973831Z 
2022-11-23T01:33:43.6974023Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6974095Z Ran 1 test in 2.274s
2022-11-23T01:33:43.6974100Z 
2022-11-23T01:33:43.6974159Z OK
2022-11-23T01:33:43.6974164Z 
2022-11-23T01:33:43.6974248Z Generating XML reports...
2022-11-23T01:33:43.6974588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013110.xml
2022-11-23T01:33:43.6974817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6975205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6975317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6975596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6975732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6975737Z 
2022-11-23T01:33:43.6975807Z Running tests...
2022-11-23T01:33:43.6975999Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6976236Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20500
2022-11-23T01:33:43.6976390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20501
2022-11-23T01:33:43.6976543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20502
2022-11-23T01:33:43.6976817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6976928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6977211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6977341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6977518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6977863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6978139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6978260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6978539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6978659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6978890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6979162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6979283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6979566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6979698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6979873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6980170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6980468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6980620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6980781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6980941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6981114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6981286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6981458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6981752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6982046Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6982340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6982574Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6982815Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6983050Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6983298Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6983542Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6983801Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6984221Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6984229Z 
2022-11-23T01:33:43.6984483Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6984749Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6984985Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6985208Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6985452Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6985703Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6985981Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6986222Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6986474Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6986728Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6986970Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6987208Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6987431Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6987674Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6987929Z STAGE:2022-11-23 01:31:16 20500:20500 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6988338Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6988344Z 
2022-11-23T01:33:43.6988598Z STAGE:2022-11-23 01:31:16 20502:20502 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6988852Z STAGE:2022-11-23 01:31:16 20501:20501 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6988918Z ok (2.376s)
2022-11-23T01:33:43.6988923Z 
2022-11-23T01:33:43.6989117Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6989190Z Ran 1 test in 2.376s
2022-11-23T01:33:43.6989195Z 
2022-11-23T01:33:43.6989242Z OK
2022-11-23T01:33:43.6989258Z 
2022-11-23T01:33:43.6989331Z Generating XML reports...
2022-11-23T01:33:43.6989670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013114.xml
2022-11-23T01:33:43.6989904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.6990180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6990302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6990582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6990714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6990719Z 
2022-11-23T01:33:43.6990789Z Running tests...
2022-11-23T01:33:43.6990969Z ----------------------------------------------------------------------
2022-11-23T01:33:43.6991207Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20565
2022-11-23T01:33:43.6991365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20566
2022-11-23T01:33:43.6991517Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20567
2022-11-23T01:33:43.6991817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6991940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6992223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6992356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6992532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.6992790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6992941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6993222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6993355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6993532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.6993801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.6993921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.6994198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.6994319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.6994498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.6994790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6995088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6995384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.6995548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.6995708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.6995868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.6996040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.6996201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.6996372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.6996666Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6996961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6997252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.6997501Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6997739Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6997975Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6998223Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6998468Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6998740Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6998985Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.6999240Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6999491Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.6999727Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.6999995Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7000227Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7000472Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7000715Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7000956Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7001196Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7001449Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7001699Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7001942Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7002179Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7002415Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7002658Z STAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7002885Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7003305Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:20 20565:20565 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7003310Z 
2022-11-23T01:33:43.7003568Z STAGE:2022-11-23 01:31:20 20566:20566 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7003822Z STAGE:2022-11-23 01:31:20 20567:20567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7003889Z ok (2.279s)
2022-11-23T01:33:43.7003895Z 
2022-11-23T01:33:43.7004088Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7004163Z Ran 1 test in 2.279s
2022-11-23T01:33:43.7004169Z 
2022-11-23T01:33:43.7004227Z OK
2022-11-23T01:33:43.7004232Z 
2022-11-23T01:33:43.7004315Z Generating XML reports...
2022-11-23T01:33:43.7004654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013118.xml
2022-11-23T01:33:43.7004877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7005153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7005277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7005558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7005694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7005699Z 
2022-11-23T01:33:43.7005770Z Running tests...
2022-11-23T01:33:43.7005994Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7006229Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20630
2022-11-23T01:33:43.7006373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20631
2022-11-23T01:33:43.7006525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20632
2022-11-23T01:33:43.7006798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7006948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7007230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7007364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7007542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7007890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7008012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7008279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7008410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7008587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7008859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7008979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7009260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7009390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7009566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7009867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7010150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7010449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7010612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7010775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7010937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7011108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7011281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7011451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7011746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7012030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7012321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7012910Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7012918Z 
2022-11-23T01:33:43.7012923Z 
2022-11-23T01:33:43.7013171Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7013416Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7013672Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7013945Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7014197Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7014450Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7014692Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7015009Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7015244Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7015490Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7015732Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7015982Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7016402Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7016407Z 
2022-11-23T01:33:43.7016663Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7016904Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7017145Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7017381Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7017614Z STAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7017857Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7018275Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:24 20630:20630 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7018280Z 
2022-11-23T01:33:43.7018536Z STAGE:2022-11-23 01:31:24 20631:20631 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7018791Z STAGE:2022-11-23 01:31:24 20632:20632 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7018859Z ok (2.273s)
2022-11-23T01:33:43.7018863Z 
2022-11-23T01:33:43.7019058Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7019134Z Ran 1 test in 2.273s
2022-11-23T01:33:43.7019139Z 
2022-11-23T01:33:43.7019198Z OK
2022-11-23T01:33:43.7019203Z 
2022-11-23T01:33:43.7019284Z Generating XML reports...
2022-11-23T01:33:43.7019613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml
2022-11-23T01:33:43.7019891Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7020167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7020290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7020570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7020703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7020708Z 
2022-11-23T01:33:43.7020780Z Running tests...
2022-11-23T01:33:43.7021006Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7021224Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20695
2022-11-23T01:33:43.7021380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20696
2022-11-23T01:33:43.7021534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20697
2022-11-23T01:33:43.7021810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7021931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7022211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7022344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7022520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7022797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7022906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7023188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7023320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7023494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7023767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7023887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7024164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7024298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7024459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7024756Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7025052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7025348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7025513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7025675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7025847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7026021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7026182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7026365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7026659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7026953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7027246Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7027493Z STAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7027776Z STAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7028187Z STAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7028193Z 
2022-11-23T01:33:43.7028454Z STAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7028712Z STAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7028953Z STAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7029189Z STAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7029592Z STAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7029610Z 
2022-11-23T01:33:43.7029855Z STAGE:2022-11-23 01:31:28 20697:20697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7030111Z STAGE:2022-11-23 01:31:28 20696:20696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7030179Z ok (2.276s)
2022-11-23T01:33:43.7030184Z 
2022-11-23T01:33:43.7030379Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7030452Z Ran 1 test in 2.276s
2022-11-23T01:33:43.7030457Z 
2022-11-23T01:33:43.7030516Z OK
2022-11-23T01:33:43.7030520Z 
2022-11-23T01:33:43.7030604Z Generating XML reports...
2022-11-23T01:33:43.7030943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013126.xml
2022-11-23T01:33:43.7031163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7031440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7031564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7031848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7031984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7031989Z 
2022-11-23T01:33:43.7032062Z Running tests...
2022-11-23T01:33:43.7032256Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7032483Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20756
2022-11-23T01:33:43.7032641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20757
2022-11-23T01:33:43.7032784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20758
2022-11-23T01:33:43.7033054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7033175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7033481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7033614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7033792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7034065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7034185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7034451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7034611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7034785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7035053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7058617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7059135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7059325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7059569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7059966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7060275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7060569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7060733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7060884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7061038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7061213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7061385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7061556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7061857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7062153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7062404Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7062696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7062931Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7063176Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7063414Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7063668Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7063918Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7064269Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7064501Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7064734Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7064965Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7065204Z STAGE:2022-11-23 01:31:32 20757:20757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7065453Z STAGE:2022-11-23 01:31:32 20758:20758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7065558Z ok (2.271s)
2022-11-23T01:33:43.7065564Z 
2022-11-23T01:33:43.7065757Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7065825Z Ran 1 test in 2.271s
2022-11-23T01:33:43.7065831Z 
2022-11-23T01:33:43.7065884Z OK
2022-11-23T01:33:43.7065889Z 
2022-11-23T01:33:43.7065967Z Generating XML reports...
2022-11-23T01:33:43.7066308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013130.xml
2022-11-23T01:33:43.7066529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7066806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7066984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7067288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7067420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7067425Z 
2022-11-23T01:33:43.7067490Z Running tests...
2022-11-23T01:33:43.7067681Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7067917Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20817
2022-11-23T01:33:43.7068068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20818
2022-11-23T01:33:43.7068210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20819
2022-11-23T01:33:43.7068481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7068596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7068870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7069000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7069174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7069441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7069556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7069835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7069954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7070124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7070391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7070510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7070785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7070912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7071153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7071447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7071729Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7072021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7072216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7072379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7072555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7072731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7072890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7073062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7073356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7073632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7073924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7074173Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7074418Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7074669Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7074911Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7075168Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7075423Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7075662Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7075887Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7076128Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7076368Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7076622Z STAGE:2022-11-23 01:31:36 20819:20819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7076873Z STAGE:2022-11-23 01:31:36 20818:20818 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7076944Z ok (2.376s)
2022-11-23T01:33:43.7076949Z 
2022-11-23T01:33:43.7077145Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7077221Z Ran 1 test in 2.376s
2022-11-23T01:33:43.7077226Z 
2022-11-23T01:33:43.7077285Z OK
2022-11-23T01:33:43.7077291Z 
2022-11-23T01:33:43.7077362Z Generating XML reports...
2022-11-23T01:33:43.7077706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013134.xml
2022-11-23T01:33:43.7077939Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7078242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7078366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7078649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7078785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7078790Z 
2022-11-23T01:33:43.7078862Z Running tests...
2022-11-23T01:33:43.7079057Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7079310Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20878
2022-11-23T01:33:43.7079468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20879
2022-11-23T01:33:43.7079622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20880
2022-11-23T01:33:43.7079898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7080019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7080299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7080433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7080607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7080866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7080991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7081269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7081407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7081581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7081848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7081963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7082235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7082362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7082527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7082821Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7083112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7083396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7083552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7083708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7083873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7084042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7084194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7084351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7084666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7084950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7085232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7085474Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7085707Z STAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7085974Z STAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7086208Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7086633Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7086639Z 
2022-11-23T01:33:43.7086874Z STAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7087098Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7087502Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7087510Z 
2022-11-23T01:33:43.7088018Z STAGE:2022-11-23 01:31:40 20879:20879 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:31:40 20880:20880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7088024Z 
2022-11-23T01:33:43.7088084Z ok (2.374s)
2022-11-23T01:33:43.7088089Z 
2022-11-23T01:33:43.7088283Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7088351Z Ran 1 test in 2.375s
2022-11-23T01:33:43.7088356Z 
2022-11-23T01:33:43.7088407Z OK
2022-11-23T01:33:43.7088412Z 
2022-11-23T01:33:43.7088490Z Generating XML reports...
2022-11-23T01:33:43.7088827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013138.xml
2022-11-23T01:33:43.7089056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7089321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7089440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7089716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7089845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7089850Z 
2022-11-23T01:33:43.7089915Z Running tests...
2022-11-23T01:33:43.7090107Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7090325Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20939
2022-11-23T01:33:43.7090474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20940
2022-11-23T01:33:43.7090622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20941
2022-11-23T01:33:43.7090886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7091000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7091272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7091434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7091607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7091874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7091992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7092266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7092417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7092586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7092877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7093142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7093255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7093527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7093653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7093820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7094114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7094401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7094559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7094717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7095089Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7095244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7095482Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7095714Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7095958Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7096194Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7096438Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7096679Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7096928Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7097173Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7097408Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7097637Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7097867Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7098105Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7098352Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7098634Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7098873Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7099122Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7099366Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7099598Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7099876Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7100103Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7100342Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7100579Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7100804Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7101048Z STAGE:2022-11-23 01:31:44 20939:20939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7101293Z STAGE:2022-11-23 01:31:44 20940:20940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7101539Z STAGE:2022-11-23 01:31:44 20941:20941 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7101601Z ok (2.172s)
2022-11-23T01:33:43.7101606Z 
2022-11-23T01:33:43.7101793Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7101861Z Ran 1 test in 2.173s
2022-11-23T01:33:43.7101866Z 
2022-11-23T01:33:43.7101919Z OK
2022-11-23T01:33:43.7101923Z 
2022-11-23T01:33:43.7101996Z Generating XML reports...
2022-11-23T01:33:43.7102335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013142.xml
2022-11-23T01:33:43.7102562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7102831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7102947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7103222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7103353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7103358Z 
2022-11-23T01:33:43.7103422Z Running tests...
2022-11-23T01:33:43.7103612Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7103826Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20995
2022-11-23T01:33:43.7103976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20996
2022-11-23T01:33:43.7104125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20997
2022-11-23T01:33:43.7104390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7104506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7104779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7104908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7105077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7105374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7105485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7105759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7105888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7106060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7106325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7106467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7106738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7106867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7107032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7107325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7107615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7107901Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7108061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7108301Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7108457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7108615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7109010Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7109016Z 
2022-11-23T01:33:43.7109255Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7109661Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7109674Z 
2022-11-23T01:33:43.7109909Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7110161Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7110415Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7110649Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7110878Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7111105Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7111342Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7111579Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7111824Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7112065Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7112323Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7112568Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7112962Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7112966Z 
2022-11-23T01:33:43.7113200Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7113467Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7113714Z STAGE:2022-11-23 01:31:48 20996:20996 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7113951Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7114188Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7114434Z STAGE:2022-11-23 01:31:48 20995:20995 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7114672Z STAGE:2022-11-23 01:31:48 20997:20997 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7114733Z ok (2.273s)
2022-11-23T01:33:43.7114738Z 
2022-11-23T01:33:43.7114927Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7114994Z Ran 1 test in 2.273s
2022-11-23T01:33:43.7115001Z 
2022-11-23T01:33:43.7115053Z OK
2022-11-23T01:33:43.7115057Z 
2022-11-23T01:33:43.7115133Z Generating XML reports...
2022-11-23T01:33:43.7115470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013146.xml
2022-11-23T01:33:43.7115702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7115965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7116082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7116355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7116483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7116488Z 
2022-11-23T01:33:43.7116552Z Running tests...
2022-11-23T01:33:43.7116744Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7116938Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T01:33:43.7116944Z 
2022-11-23T01:33:43.7117132Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7117199Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7117206Z 
2022-11-23T01:33:43.7117265Z OK (skipped=1)
2022-11-23T01:33:43.7117269Z 
2022-11-23T01:33:43.7117344Z Generating XML reports...
2022-11-23T01:33:43.7117680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013150.xml
2022-11-23T01:33:43.7117907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7118178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7118294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7118568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7118697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7118701Z 
2022-11-23T01:33:43.7118765Z Running tests...
2022-11-23T01:33:43.7118974Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7119202Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21061
2022-11-23T01:33:43.7119352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21062
2022-11-23T01:33:43.7119498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21063
2022-11-23T01:33:43.7119767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7119916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7120187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7120315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7120482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7120746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7120862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7121132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7121260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7121432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7121729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7121992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7122109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7122375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7122499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7122667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7122960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7123250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7123410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7123564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7123722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7123963Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7124190Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7124419Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7124657Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7124891Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7125143Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7125380Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7125656Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7125901Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7126134Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7126358Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7126593Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7126860Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7127106Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7127340Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7127576Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7127905Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7128158Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7128397Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7128620Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7128854Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7129087Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7129495Z STAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7129501Z 
2022-11-23T01:33:43.7129747Z STAGE:2022-11-23 01:31:53 21061:21061 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7130169Z STAGE:2022-11-23 01:31:53 21063:21063 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:31:53 21062:21062 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7130174Z 
2022-11-23T01:33:43.7130235Z ok (2.176s)
2022-11-23T01:33:43.7130240Z 
2022-11-23T01:33:43.7130433Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7130502Z Ran 1 test in 2.176s
2022-11-23T01:33:43.7130507Z 
2022-11-23T01:33:43.7130564Z OK
2022-11-23T01:33:43.7130569Z 
2022-11-23T01:33:43.7130640Z Generating XML reports...
2022-11-23T01:33:43.7130982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013152.xml
2022-11-23T01:33:43.7131211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7131482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7131604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7131876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7132003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7132011Z 
2022-11-23T01:33:43.7132073Z Running tests...
2022-11-23T01:33:43.7132254Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7132462Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T01:33:43.7132468Z 
2022-11-23T01:33:43.7132687Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7132755Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7132760Z 
2022-11-23T01:33:43.7132822Z OK (skipped=1)
2022-11-23T01:33:43.7132826Z 
2022-11-23T01:33:43.7132901Z Generating XML reports...
2022-11-23T01:33:43.7133233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml
2022-11-23T01:33:43.7133458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7133725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7133862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7134140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7134275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7134280Z 
2022-11-23T01:33:43.7134350Z Running tests...
2022-11-23T01:33:43.7134543Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7134743Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2022-11-23T01:33:43.7134748Z 
2022-11-23T01:33:43.7135051Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7135126Z Ran 1 test in 0.003s
2022-11-23T01:33:43.7135132Z 
2022-11-23T01:33:43.7135201Z OK (skipped=1)
2022-11-23T01:33:43.7135208Z 
2022-11-23T01:33:43.7135277Z Generating XML reports...
2022-11-23T01:33:43.7135618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml
2022-11-23T01:33:43.7135849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7136125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7136250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7136530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7136663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7136668Z 
2022-11-23T01:33:43.7136738Z Running tests...
2022-11-23T01:33:43.7136934Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7137185Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21137
2022-11-23T01:33:43.7137344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21138
2022-11-23T01:33:43.7137499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21139
2022-11-23T01:33:43.7137772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7137894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7138172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7138303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7138477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7138737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7138858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7139139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7139319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7139498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7139768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7139890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7140170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7140304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7140503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7140804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7141105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7141396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7141560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7141722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7141972Z STAGE:2022-11-23 01:32:00 21139:21139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7142213Z STAGE:2022-11-23 01:32:00 21138:21138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7142462Z STAGE:2022-11-23 01:32:00 21139:21139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7142707Z STAGE:2022-11-23 01:32:00 21139:21139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7142951Z STAGE:2022-11-23 01:32:00 21139:21139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7143113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7143357Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7143599Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7143840Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7144097Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7144352Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7144592Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7144817Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7145058Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7145298Z STAGE:2022-11-23 01:32:01 21139:21139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7145716Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7145723Z 
2022-11-23T01:33:43.7146154Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:01 21139:21139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7146160Z 
2022-11-23T01:33:43.7146400Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7146661Z STAGE:2022-11-23 01:32:01 21139:21139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7146896Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7147139Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7147381Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7147797Z STAGE:2022-11-23 01:32:01 21138:21138 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:01 21139:21139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7148080Z STAGE:2022-11-23 01:32:01 21137:21137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7148085Z 
2022-11-23T01:33:43.7148341Z STAGE:2022-11-23 01:32:01 21139:21139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7148407Z ok (2.273s)
2022-11-23T01:33:43.7148412Z 
2022-11-23T01:33:43.7148605Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7148680Z Ran 1 test in 2.273s
2022-11-23T01:33:43.7148685Z 
2022-11-23T01:33:43.7148743Z OK
2022-11-23T01:33:43.7148748Z 
2022-11-23T01:33:43.7148830Z Generating XML reports...
2022-11-23T01:33:43.7149172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013159.xml
2022-11-23T01:33:43.7149404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7149669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7149792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7150075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7150209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7150214Z 
2022-11-23T01:33:43.7150284Z Running tests...
2022-11-23T01:33:43.7150481Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7150667Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:33:43.7150672Z 
2022-11-23T01:33:43.7150864Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7150927Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7150946Z 
2022-11-23T01:33:43.7151004Z OK (skipped=1)
2022-11-23T01:33:43.7151009Z 
2022-11-23T01:33:43.7151092Z Generating XML reports...
2022-11-23T01:33:43.7151429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013203.xml
2022-11-23T01:33:43.7151663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7151938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7152061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7152344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7152480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7152485Z 
2022-11-23T01:33:43.7152543Z Running tests...
2022-11-23T01:33:43.7152736Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7152927Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:33:43.7152931Z 
2022-11-23T01:33:43.7153123Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7153224Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7153229Z 
2022-11-23T01:33:43.7153300Z OK (skipped=1)
2022-11-23T01:33:43.7153304Z 
2022-11-23T01:33:43.7153386Z Generating XML reports...
2022-11-23T01:33:43.7153725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013204.xml
2022-11-23T01:33:43.7153956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7154217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7154365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7154647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7154779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7154784Z 
2022-11-23T01:33:43.7154852Z Running tests...
2022-11-23T01:33:43.7155043Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7155272Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21213
2022-11-23T01:33:43.7155428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21214
2022-11-23T01:33:43.7155569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21215
2022-11-23T01:33:43.7155842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7155965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7156246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7156378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7156555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7156826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7156948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7157225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7157346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7157523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7157796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7157917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7158198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7158332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7158509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7158806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7159102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7159382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7159546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7159709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7159983Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7160224Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7160386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7160630Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7160877Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7161317Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7161323Z 
2022-11-23T01:33:43.7161925Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7161931Z 
2022-11-23T01:33:43.7161935Z 
2022-11-23T01:33:43.7162332Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7162338Z 
2022-11-23T01:33:43.7162580Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7162817Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7163072Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7163484Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7163489Z 
2022-11-23T01:33:43.7163742Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7163994Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7164234Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7164469Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7164704Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7164949Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7165190Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7165417Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7165669Z STAGE:2022-11-23 01:32:08 21213:21213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7165922Z STAGE:2022-11-23 01:32:08 21215:21215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7166173Z STAGE:2022-11-23 01:32:08 21214:21214 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7166240Z ok (2.273s)
2022-11-23T01:33:43.7166247Z 
2022-11-23T01:33:43.7166446Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7166521Z Ran 1 test in 2.273s
2022-11-23T01:33:43.7166525Z 
2022-11-23T01:33:43.7166584Z OK
2022-11-23T01:33:43.7166589Z 
2022-11-23T01:33:43.7166659Z Generating XML reports...
2022-11-23T01:33:43.7167033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013206.xml
2022-11-23T01:33:43.7167269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7167542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7167666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7168036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7168171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7168210Z 
2022-11-23T01:33:43.7168281Z Running tests...
2022-11-23T01:33:43.7168478Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7168688Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21269
2022-11-23T01:33:43.7168846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21270
2022-11-23T01:33:43.7168999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21271
2022-11-23T01:33:43.7169272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7169396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7169678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7169816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7169993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7170266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7170377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7170658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7170791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7170965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7171235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7171356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7171636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7171771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7171935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7172234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7172532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7172824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7172987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7173150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7173310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7173559Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7173824Z STAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7174048Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7174295Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7174551Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7175075Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7175168Z 
2022-11-23T01:33:43.7175604Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7175610Z 
2022-11-23T01:33:43.7175851Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7176252Z STAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7176257Z 
2022-11-23T01:33:43.7176823Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7176830Z 
2022-11-23T01:33:43.7176834Z 
2022-11-23T01:33:43.7177436Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7177441Z 
2022-11-23T01:33:43.7177444Z 
2022-11-23T01:33:43.7177686Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7178082Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7178087Z 
2022-11-23T01:33:43.7178331Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7178590Z STAGE:2022-11-23 01:32:12 21271:21271 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7178997Z STAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7179002Z 
2022-11-23T01:33:43.7179431Z STAGE:2022-11-23 01:32:12 21270:21270 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:12 21269:21269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7179436Z 
2022-11-23T01:33:43.7179502Z ok (2.174s)
2022-11-23T01:33:43.7179507Z 
2022-11-23T01:33:43.7179702Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7179763Z Ran 1 test in 2.174s
2022-11-23T01:33:43.7179767Z 
2022-11-23T01:33:43.7179827Z OK
2022-11-23T01:33:43.7179832Z 
2022-11-23T01:33:43.7179917Z Generating XML reports...
2022-11-23T01:33:43.7180258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013210.xml
2022-11-23T01:33:43.7180492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7180801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7180925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7181206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7181341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7181345Z 
2022-11-23T01:33:43.7181457Z Running tests...
2022-11-23T01:33:43.7181651Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7181878Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21325
2022-11-23T01:33:43.7182062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21326
2022-11-23T01:33:43.7182218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21327
2022-11-23T01:33:43.7182493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7182614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7182894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7183016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7183193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7183464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7183588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7183865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7183997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7184174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7184446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7184571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7184837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7184971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7185147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7185450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7185750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7186045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7186210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7186370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7186533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7186587Z ok (2.178s)
2022-11-23T01:33:43.7186593Z 
2022-11-23T01:33:43.7186787Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7186860Z Ran 1 test in 2.178s
2022-11-23T01:33:43.7186865Z 
2022-11-23T01:33:43.7186928Z OK
2022-11-23T01:33:43.7186933Z 
2022-11-23T01:33:43.7187015Z Generating XML reports...
2022-11-23T01:33:43.7187383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml
2022-11-23T01:33:43.7187617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7187890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7188000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7188278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7188411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7188445Z 
2022-11-23T01:33:43.7188517Z Running tests...
2022-11-23T01:33:43.7188712Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7188943Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21378
2022-11-23T01:33:43.7189100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21379
2022-11-23T01:33:43.7189250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21380
2022-11-23T01:33:43.7189522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7189632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7189913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7190048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7190224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7190497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7190619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7190896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7191028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7191189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7191463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7191585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7191865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7191998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7192172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7192471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7192766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7193056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7193206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7193370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7193531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7193782Z STAGE:2022-11-23 01:32:19 21378:21378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7194051Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7194290Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7194537Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7194794Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7195041Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7195269Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7195565Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7195817Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7196060Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7196298Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7196532Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7196775Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7197028Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7197445Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7197451Z 
2022-11-23T01:33:43.7197880Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7197886Z 
2022-11-23T01:33:43.7198123Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7198346Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7198580Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7198825Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7199080Z STAGE:2022-11-23 01:32:20 21378:21378 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7199324Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7199566Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7199820Z STAGE:2022-11-23 01:32:20 21379:21379 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7200070Z STAGE:2022-11-23 01:32:20 21380:21380 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7200135Z ok (2.173s)
2022-11-23T01:33:43.7200140Z 
2022-11-23T01:33:43.7200324Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7200397Z Ran 1 test in 2.173s
2022-11-23T01:33:43.7200402Z 
2022-11-23T01:33:43.7200461Z OK
2022-11-23T01:33:43.7200465Z 
2022-11-23T01:33:43.7200551Z Generating XML reports...
2022-11-23T01:33:43.7200893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013218.xml
2022-11-23T01:33:43.7201126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7201426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7201551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7201818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7201953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7201958Z 
2022-11-23T01:33:43.7202029Z Running tests...
2022-11-23T01:33:43.7202224Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7202409Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-11-23T01:33:43.7202440Z 
2022-11-23T01:33:43.7202632Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7202705Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7202710Z 
2022-11-23T01:33:43.7202779Z OK (skipped=1)
2022-11-23T01:33:43.7202784Z 
2022-11-23T01:33:43.7202866Z Generating XML reports...
2022-11-23T01:33:43.7203197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013222.xml
2022-11-23T01:33:43.7203428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7203700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7203822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7204101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7204237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7204242Z 
2022-11-23T01:33:43.7204313Z Running tests...
2022-11-23T01:33:43.7204504Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7204687Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:33:43.7204706Z 
2022-11-23T01:33:43.7204880Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7204954Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7204959Z 
2022-11-23T01:33:43.7205028Z OK (skipped=1)
2022-11-23T01:33:43.7205032Z 
2022-11-23T01:33:43.7205115Z Generating XML reports...
2022-11-23T01:33:43.7205448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml
2022-11-23T01:33:43.7205681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7205955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7206076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7206345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7206480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7206484Z 
2022-11-23T01:33:43.7206554Z Running tests...
2022-11-23T01:33:43.7206744Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7206976Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21454
2022-11-23T01:33:43.7207131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21455
2022-11-23T01:33:43.7207288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21456
2022-11-23T01:33:43.7207561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7207747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7208051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7208188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7208366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7208639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7208760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7209038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7209203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7209380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7209643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7209764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7210043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7210175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7210349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7210648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7210943Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7211232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7211399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7211552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7211713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7211887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7212060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7212359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7212532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7212824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7213120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7213367Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7213608Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7213832Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7214243Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7214251Z 
2022-11-23T01:33:43.7214498Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7215091Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7215098Z 
2022-11-23T01:33:43.7215357Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7215598Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7215833Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7216067Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7216347Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7216602Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7216835Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7217076Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7217329Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7217579Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7217819Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7218057Z STAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7218289Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7218702Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7218708Z 
2022-11-23T01:33:43.7218952Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7219377Z STAGE:2022-11-23 01:32:27 21456:21456 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:32:27 21454:21454 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7219383Z 
2022-11-23T01:33:43.7219634Z STAGE:2022-11-23 01:32:27 21455:21455 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7219690Z ok (2.271s)
2022-11-23T01:33:43.7219695Z 
2022-11-23T01:33:43.7219890Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7219963Z Ran 1 test in 2.272s
2022-11-23T01:33:43.7219968Z 
2022-11-23T01:33:43.7220027Z OK
2022-11-23T01:33:43.7220031Z 
2022-11-23T01:33:43.7220114Z Generating XML reports...
2022-11-23T01:33:43.7220457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013225.xml
2022-11-23T01:33:43.7220692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7220965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7221075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7221355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7221491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7221495Z 
2022-11-23T01:33:43.7221569Z Running tests...
2022-11-23T01:33:43.7221764Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7222019Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21519
2022-11-23T01:33:43.7222178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21520
2022-11-23T01:33:43.7222331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21521
2022-11-23T01:33:43.7222603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7222710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7222988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7223147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7223326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7223597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7223720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7224001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7224138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7224301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7224572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7224694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7224972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7225105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7225281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7225580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7225877Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7226172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7226326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7226491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7226653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7226826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:33:43.7227000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:33:43.7227173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:33:43.7227469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7227763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7228012Z STAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7228243Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7228491Z STAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7228758Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7229019Z STAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7229273Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7229510Z STAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7229743Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7230154Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7230195Z 
2022-11-23T01:33:43.7230451Z STAGE:2022-11-23 01:32:31 21521:21521 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7230706Z STAGE:2022-11-23 01:32:31 21520:21520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7230992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-11-23T01:33:43.7231058Z ok (2.374s)
2022-11-23T01:33:43.7231064Z 
2022-11-23T01:33:43.7231260Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7231334Z Ran 1 test in 2.374s
2022-11-23T01:33:43.7231339Z 
2022-11-23T01:33:43.7231398Z OK
2022-11-23T01:33:43.7231403Z 
2022-11-23T01:33:43.7231485Z Generating XML reports...
2022-11-23T01:33:43.7231831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013229.xml
2022-11-23T01:33:43.7232064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7232340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7232453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7232733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7232868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7232873Z 
2022-11-23T01:33:43.7232945Z Running tests...
2022-11-23T01:33:43.7233139Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7233373Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21580
2022-11-23T01:33:43.7233530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21581
2022-11-23T01:33:43.7233684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21582
2022-11-23T01:33:43.7233946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7234069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7234349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7234481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7234658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7234926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7235052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7235330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7235464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7235660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7235933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7236054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7236332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7236465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7236666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7236965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7237261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7237553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7237703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7237867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7238029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7238095Z ok (2.174s)
2022-11-23T01:33:43.7238100Z 
2022-11-23T01:33:43.7238294Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7238370Z Ran 1 test in 2.174s
2022-11-23T01:33:43.7238375Z 
2022-11-23T01:33:43.7238434Z OK
2022-11-23T01:33:43.7238438Z 
2022-11-23T01:33:43.7238520Z Generating XML reports...
2022-11-23T01:33:43.7238850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013233.xml
2022-11-23T01:33:43.7239085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7239356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7239479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7239758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7239892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7239899Z 
2022-11-23T01:33:43.7239970Z Running tests...
2022-11-23T01:33:43.7240167Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7240389Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21633
2022-11-23T01:33:43.7240534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21634
2022-11-23T01:33:43.7240689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21635
2022-11-23T01:33:43.7240961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7241083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7241363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7241496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7241674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7241947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7242055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7242363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7242499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7242675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7242946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7243069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7243390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7243522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7243698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7243987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7244286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7244576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7244741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7244904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7245067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7245133Z ok (2.072s)
2022-11-23T01:33:43.7245138Z 
2022-11-23T01:33:43.7245333Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7245406Z Ran 1 test in 2.073s
2022-11-23T01:33:43.7245411Z 
2022-11-23T01:33:43.7245460Z OK
2022-11-23T01:33:43.7245465Z 
2022-11-23T01:33:43.7245547Z Generating XML reports...
2022-11-23T01:33:43.7245886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013237.xml
2022-11-23T01:33:43.7246122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7246393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7246517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7246796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7246931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7246935Z 
2022-11-23T01:33:43.7246994Z Running tests...
2022-11-23T01:33:43.7247191Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7247424Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21686
2022-11-23T01:33:43.7247577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21687
2022-11-23T01:33:43.7247828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21688
2022-11-23T01:33:43.7248105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7248227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7248511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7248645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7248842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7249116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7249239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7249521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7249656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7249834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7250133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7250255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7250533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7250655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7250832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7251131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7251430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7251720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7251887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7252050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7252212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7252266Z ok (2.174s)
2022-11-23T01:33:43.7252283Z 
2022-11-23T01:33:43.7252466Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7252539Z Ran 1 test in 2.174s
2022-11-23T01:33:43.7252544Z 
2022-11-23T01:33:43.7252604Z OK
2022-11-23T01:33:43.7252608Z 
2022-11-23T01:33:43.7252691Z Generating XML reports...
2022-11-23T01:33:43.7253032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013241.xml
2022-11-23T01:33:43.7253265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7253540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7253663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7253931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7254066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7254072Z 
2022-11-23T01:33:43.7254141Z Running tests...
2022-11-23T01:33:43.7254337Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7254590Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21739
2022-11-23T01:33:43.7254748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21740
2022-11-23T01:33:43.7255033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21741
2022-11-23T01:33:43.7255312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7255435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7255750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7255885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7256060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7256332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7256453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7256733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7256905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7257079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7257340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7257463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7257744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7257878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7258054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7258351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7258649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7258938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7259104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7259256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7259506Z STAGE:2022-11-23 01:32:46 21741:21741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7259745Z STAGE:2022-11-23 01:32:46 21740:21740 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7259909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7260153Z STAGE:2022-11-23 01:32:46 21739:21739 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7260402Z STAGE:2022-11-23 01:32:46 21739:21739 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7260659Z STAGE:2022-11-23 01:32:46 21739:21739 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7260908Z STAGE:2022-11-23 01:32:46 21740:21740 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7261150Z STAGE:2022-11-23 01:32:46 21741:21741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7261394Z STAGE:2022-11-23 01:32:46 21740:21740 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7261646Z STAGE:2022-11-23 01:32:46 21741:21741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7261713Z ok (2.273s)
2022-11-23T01:33:43.7261718Z 
2022-11-23T01:33:43.7261914Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7261990Z Ran 1 test in 2.273s
2022-11-23T01:33:43.7261995Z 
2022-11-23T01:33:43.7262052Z OK
2022-11-23T01:33:43.7262057Z 
2022-11-23T01:33:43.7262141Z Generating XML reports...
2022-11-23T01:33:43.7262483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013245.xml
2022-11-23T01:33:43.7262744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7263005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7263129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7263410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7263544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7263550Z 
2022-11-23T01:33:43.7263649Z Running tests...
2022-11-23T01:33:43.7263843Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7264095Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21795
2022-11-23T01:33:43.7264251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21796
2022-11-23T01:33:43.7264392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21797
2022-11-23T01:33:43.7264664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7264786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7265066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7265201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7265380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7265654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7265775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7266057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7266177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7266353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7266653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7266924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7267049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7267329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7267463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7267639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7267936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7268217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7268382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7268631Z STAGE:2022-11-23 01:32:50 21796:21796 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7268795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7268958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7269203Z STAGE:2022-11-23 01:32:50 21795:21795 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7269469Z STAGE:2022-11-23 01:32:50 21797:21797 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7269718Z STAGE:2022-11-23 01:32:50 21795:21795 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7269976Z STAGE:2022-11-23 01:32:50 21795:21795 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7270209Z STAGE:2022-11-23 01:32:50 21796:21796 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7270451Z STAGE:2022-11-23 01:32:50 21797:21797 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7270733Z STAGE:2022-11-23 01:32:50 21796:21796 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7270987Z STAGE:2022-11-23 01:32:50 21797:21797 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7271053Z ok (2.276s)
2022-11-23T01:33:43.7271058Z 
2022-11-23T01:33:43.7271254Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7271328Z Ran 1 test in 2.277s
2022-11-23T01:33:43.7271333Z 
2022-11-23T01:33:43.7271390Z OK
2022-11-23T01:33:43.7271394Z 
2022-11-23T01:33:43.7271465Z Generating XML reports...
2022-11-23T01:33:43.7271809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml
2022-11-23T01:33:43.7272043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7272317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7272445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7272724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7272858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7272866Z 
2022-11-23T01:33:43.7272936Z Running tests...
2022-11-23T01:33:43.7273131Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7273362Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21851
2022-11-23T01:33:43.7273519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21852
2022-11-23T01:33:43.7273673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21853
2022-11-23T01:33:43.7273948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7274070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7274346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7274482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7274660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7274922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7275044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7275325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7275460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7275638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7275908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7276031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7276337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7276473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7276636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7276935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7277233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7277549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7277714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7277965Z STAGE:2022-11-23 01:32:54 21852:21852 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7278129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7278291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7278535Z STAGE:2022-11-23 01:32:55 21851:21851 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7278760Z STAGE:2022-11-23 01:32:55 21853:21853 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7279010Z STAGE:2022-11-23 01:32:55 21853:21853 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7279269Z STAGE:2022-11-23 01:32:55 21853:21853 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7279517Z STAGE:2022-11-23 01:32:55 21851:21851 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7279759Z STAGE:2022-11-23 01:32:55 21852:21852 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7280014Z STAGE:2022-11-23 01:32:55 21851:21851 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7280266Z STAGE:2022-11-23 01:32:55 21852:21852 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7280331Z ok (2.279s)
2022-11-23T01:33:43.7280337Z 
2022-11-23T01:33:43.7280530Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7280592Z Ran 1 test in 2.279s
2022-11-23T01:33:43.7280596Z 
2022-11-23T01:33:43.7280658Z OK
2022-11-23T01:33:43.7280663Z 
2022-11-23T01:33:43.7280745Z Generating XML reports...
2022-11-23T01:33:43.7281089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013253.xml
2022-11-23T01:33:43.7281326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7281603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7281727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7282010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7282134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7282151Z 
2022-11-23T01:33:43.7282210Z Running tests...
2022-11-23T01:33:43.7282404Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7282580Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-11-23T01:33:43.7282585Z 
2022-11-23T01:33:43.7282774Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7282848Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7282853Z 
2022-11-23T01:33:43.7282923Z OK (skipped=1)
2022-11-23T01:33:43.7282927Z 
2022-11-23T01:33:43.7283067Z Generating XML reports...
2022-11-23T01:33:43.7283407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013257.xml
2022-11-23T01:33:43.7283625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7283897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7284020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7284300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7284463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7284468Z 
2022-11-23T01:33:43.7284539Z Running tests...
2022-11-23T01:33:43.7284731Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7284927Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-11-23T01:33:43.7284932Z 
2022-11-23T01:33:43.7285125Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7285186Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7285191Z 
2022-11-23T01:33:43.7285260Z OK (skipped=1)
2022-11-23T01:33:43.7285265Z 
2022-11-23T01:33:43.7285347Z Generating XML reports...
2022-11-23T01:33:43.7285687Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml
2022-11-23T01:33:43.7285922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7286195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7286316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7286598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7286719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7286736Z 
2022-11-23T01:33:43.7286793Z Running tests...
2022-11-23T01:33:43.7286988Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7287176Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:33:43.7287181Z 
2022-11-23T01:33:43.7287374Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7287450Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7287454Z 
2022-11-23T01:33:43.7287524Z OK (skipped=1)
2022-11-23T01:33:43.7287528Z 
2022-11-23T01:33:43.7287609Z Generating XML reports...
2022-11-23T01:33:43.7288029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013300.xml
2022-11-23T01:33:43.7288253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7288527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7288649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7288927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7289058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7289066Z 
2022-11-23T01:33:43.7289135Z Running tests...
2022-11-23T01:33:43.7289327Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7289564Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21937
2022-11-23T01:33:43.7289756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21938
2022-11-23T01:33:43.7289898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21939
2022-11-23T01:33:43.7290171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7290291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7290570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7290703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7290920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7291189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7291312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7291580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7291714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7291890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7292166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7292288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7292575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7292710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7292884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7293186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7293470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7293763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7293925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7294176Z STAGE:2022-11-23 01:33:03 21938:21938 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7294342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7294502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7294750Z STAGE:2022-11-23 01:33:03 21937:21937 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7295105Z STAGE:2022-11-23 01:33:03 21939:21939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7295357Z STAGE:2022-11-23 01:33:03 21939:21939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7295586Z STAGE:2022-11-23 01:33:03 21937:21937 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7295845Z STAGE:2022-11-23 01:33:03 21939:21939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7296101Z STAGE:2022-11-23 01:33:03 21937:21937 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7296349Z STAGE:2022-11-23 01:33:03 21938:21938 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7296600Z STAGE:2022-11-23 01:33:03 21938:21938 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7296668Z ok (2.373s)
2022-11-23T01:33:43.7296673Z 
2022-11-23T01:33:43.7296917Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7296994Z Ran 1 test in 2.374s
2022-11-23T01:33:43.7296999Z 
2022-11-23T01:33:43.7297058Z OK
2022-11-23T01:33:43.7297063Z 
2022-11-23T01:33:43.7297134Z Generating XML reports...
2022-11-23T01:33:43.7297475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013301.xml
2022-11-23T01:33:43.7297707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7297985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7298151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7298433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7298568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7298576Z 
2022-11-23T01:33:43.7298646Z Running tests...
2022-11-23T01:33:43.7298830Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7299062Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21993
2022-11-23T01:33:43.7299218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21994
2022-11-23T01:33:43.7299372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21995
2022-11-23T01:33:43.7299645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7299768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7300048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7300183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7300360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7300620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7300742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7301021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7301155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7301333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7301604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7301725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7302006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7302128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7302305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7302604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7302901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7303191Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7303393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7303584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7303747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7303813Z ok (2.172s)
2022-11-23T01:33:43.7303818Z 
2022-11-23T01:33:43.7304000Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7304073Z Ran 1 test in 2.173s
2022-11-23T01:33:43.7304079Z 
2022-11-23T01:33:43.7304163Z OK
2022-11-23T01:33:43.7304168Z 
2022-11-23T01:33:43.7304249Z Generating XML reports...
2022-11-23T01:33:43.7304589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013306.xml
2022-11-23T01:33:43.7304849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7305121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7305245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7305526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7305648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7305653Z 
2022-11-23T01:33:43.7305724Z Running tests...
2022-11-23T01:33:43.7305919Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7306169Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22046
2022-11-23T01:33:43.7306325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22047
2022-11-23T01:33:43.7306479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22048
2022-11-23T01:33:43.7306752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7306880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7307161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7307284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7307459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7307731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7307853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7308131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7308264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7308442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7308713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7308822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7309100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7309233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7309409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7309710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7310004Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7310324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7310489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7310651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7310888Z STAGE:2022-11-23 01:33:11 22047:22047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7311127Z STAGE:2022-11-23 01:33:11 22048:22048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7311321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7311565Z STAGE:2022-11-23 01:33:11 22046:22046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7311814Z STAGE:2022-11-23 01:33:11 22046:22046 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7312073Z STAGE:2022-11-23 01:33:11 22046:22046 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7312320Z STAGE:2022-11-23 01:33:11 22047:22047 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7312563Z STAGE:2022-11-23 01:33:11 22048:22048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7312820Z STAGE:2022-11-23 01:33:11 22047:22047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7313061Z STAGE:2022-11-23 01:33:11 22048:22048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7313130Z ok (2.271s)
2022-11-23T01:33:43.7313135Z 
2022-11-23T01:33:43.7313328Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7313403Z Ran 1 test in 2.271s
2022-11-23T01:33:43.7313408Z 
2022-11-23T01:33:43.7313467Z OK
2022-11-23T01:33:43.7313472Z 
2022-11-23T01:33:43.7313556Z Generating XML reports...
2022-11-23T01:33:43.7313902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013309.xml
2022-11-23T01:33:43.7314135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7314394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7314517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7314796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7314934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7314939Z 
2022-11-23T01:33:43.7315012Z Running tests...
2022-11-23T01:33:43.7315207Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7315458Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22102
2022-11-23T01:33:43.7315616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22103
2022-11-23T01:33:43.7315769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22104
2022-11-23T01:33:43.7316031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7316153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7316432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7316569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7316745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7317047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7317170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7317450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7317594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7317757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7318029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7318180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7318460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7318592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7318770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7319070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7319367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7319660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7319812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7320062Z STAGE:2022-11-23 01:33:15 22104:22104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7320225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7320389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7320638Z STAGE:2022-11-23 01:33:15 22102:22102 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7320876Z STAGE:2022-11-23 01:33:15 22103:22103 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:33:43.7321123Z STAGE:2022-11-23 01:33:15 22102:22102 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7321380Z STAGE:2022-11-23 01:33:15 22102:22102 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7321612Z STAGE:2022-11-23 01:33:15 22103:22103 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7321869Z STAGE:2022-11-23 01:33:15 22103:22103 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7322113Z STAGE:2022-11-23 01:33:15 22104:22104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:33:43.7322366Z STAGE:2022-11-23 01:33:15 22104:22104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:33:43.7322434Z ok (2.176s)
2022-11-23T01:33:43.7322439Z 
2022-11-23T01:33:43.7322634Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7322706Z Ran 1 test in 2.176s
2022-11-23T01:33:43.7322711Z 
2022-11-23T01:33:43.7322770Z OK
2022-11-23T01:33:43.7322775Z 
2022-11-23T01:33:43.7322857Z Generating XML reports...
2022-11-23T01:33:43.7323187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013313.xml
2022-11-23T01:33:43.7323421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7323693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7323816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7324126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7324261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7324266Z 
2022-11-23T01:33:43.7324337Z Running tests...
2022-11-23T01:33:43.7324531Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7324767Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22158
2022-11-23T01:33:43.7324911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22159
2022-11-23T01:33:43.7325098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22160
2022-11-23T01:33:43.7325373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7325495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7325775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7325909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7326085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7326359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7326468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7326748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7326885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7327062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7327333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7327455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7327806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7327942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7328116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7328403Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7328705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7328999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7329165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7329329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7329488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7329553Z ok (2.276s)
2022-11-23T01:33:43.7329558Z 
2022-11-23T01:33:43.7329753Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7329827Z Ran 1 test in 2.277s
2022-11-23T01:33:43.7329832Z 
2022-11-23T01:33:43.7329879Z OK
2022-11-23T01:33:43.7329886Z 
2022-11-23T01:33:43.7329969Z Generating XML reports...
2022-11-23T01:33:43.7330311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013317.xml
2022-11-23T01:33:43.7330545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7330850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7330975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7331255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7331390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7331394Z 
2022-11-23T01:33:43.7331453Z Running tests...
2022-11-23T01:33:43.7331648Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7331917Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22238
2022-11-23T01:33:43.7332073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22239
2022-11-23T01:33:43.7332226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22240
2022-11-23T01:33:43.7332500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7332620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7332900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7333032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7333197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7333468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7333589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7333869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7334005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7334179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7334449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7334570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7334835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7335080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7335255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7335552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7335850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7336142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7336306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7336469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7336630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7336714Z skip: CUDA is not available. (2.173s)
2022-11-23T01:33:43.7336733Z 
2022-11-23T01:33:43.7336916Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7336990Z Ran 1 test in 2.173s
2022-11-23T01:33:43.7336995Z 
2022-11-23T01:33:43.7337065Z OK (skipped=1)
2022-11-23T01:33:43.7337069Z 
2022-11-23T01:33:43.7337152Z Generating XML reports...
2022-11-23T01:33:43.7337542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013321.xml
2022-11-23T01:33:43.7337776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7338050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7338172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7338439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7338621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7338626Z 
2022-11-23T01:33:43.7338697Z Running tests...
2022-11-23T01:33:43.7338889Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7339131Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22291
2022-11-23T01:33:43.7339288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22292
2022-11-23T01:33:43.7339441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22293
2022-11-23T01:33:43.7339714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7339835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7340102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7340237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7340413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7340714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7340986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7341109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7341389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7341522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7341685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7341957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7342078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7342357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7342493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7342668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7342966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7343263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7343426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7343579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7343739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7343842Z skip: Need at least 2 CUDA devices (2.178s)
2022-11-23T01:33:43.7343847Z 
2022-11-23T01:33:43.7344070Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7344145Z Ran 1 test in 2.179s
2022-11-23T01:33:43.7344151Z 
2022-11-23T01:33:43.7344220Z OK (skipped=1)
2022-11-23T01:33:43.7344225Z 
2022-11-23T01:33:43.7344307Z Generating XML reports...
2022-11-23T01:33:43.7344645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml
2022-11-23T01:33:43.7344877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7345137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7345290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7345569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7345706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7345712Z 
2022-11-23T01:33:43.7345782Z Running tests...
2022-11-23T01:33:43.7345973Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7346211Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22344
2022-11-23T01:33:43.7346367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22345
2022-11-23T01:33:43.7346508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22346
2022-11-23T01:33:43.7346784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7346907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7347186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7347326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7347505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7347775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7347897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7348176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7348299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7348474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7348744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7348867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7349143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7349275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7349450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7349748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7350042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7350323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7350487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7350676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7350839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7351025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tkgcyzk
2022-11-23T01:33:43.7351210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd313zh8
2022-11-23T01:33:43.7351407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tkgcyzk/_remote_module_non_scriptable.py
2022-11-23T01:33:43.7351596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd313zh8/_remote_module_non_scriptable.py
2022-11-23T01:33:43.7351806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrlka4cg
2022-11-23T01:33:43.7351987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrlka4cg/_remote_module_non_scriptable.py
2022-11-23T01:33:43.7352054Z ok (2.073s)
2022-11-23T01:33:43.7352059Z 
2022-11-23T01:33:43.7352256Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7352331Z Ran 1 test in 2.073s
2022-11-23T01:33:43.7352336Z 
2022-11-23T01:33:43.7352394Z OK
2022-11-23T01:33:43.7352399Z 
2022-11-23T01:33:43.7352480Z Generating XML reports...
2022-11-23T01:33:43.7352825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013329.xml
2022-11-23T01:33:43.7353057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7353316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7353441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7353723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7353860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7353864Z 
2022-11-23T01:33:43.7353936Z Running tests...
2022-11-23T01:33:43.7354130Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7354359Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22397
2022-11-23T01:33:43.7354517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22398
2022-11-23T01:33:43.7354671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22399
2022-11-23T01:33:43.7354934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7355058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7355337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7355475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7355652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7355922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7356045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7356325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7356449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7356627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7356896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7357050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7357331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7357466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7357641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7357940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7358240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7358549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7358714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7358877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7359040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7359142Z skip: Need at least 2 CUDA devices (2.174s)
2022-11-23T01:33:43.7359148Z 
2022-11-23T01:33:43.7359342Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7359416Z Ran 1 test in 2.174s
2022-11-23T01:33:43.7359421Z 
2022-11-23T01:33:43.7359491Z OK (skipped=1)
2022-11-23T01:33:43.7359496Z 
2022-11-23T01:33:43.7359621Z Generating XML reports...
2022-11-23T01:33:43.7359954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013333.xml
2022-11-23T01:33:43.7360186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7360459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7360584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7360867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7361002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7361006Z 
2022-11-23T01:33:43.7361075Z Running tests...
2022-11-23T01:33:43.7361271Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7361517Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22450
2022-11-23T01:33:43.7361676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22451
2022-11-23T01:33:43.7361831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22452
2022-11-23T01:33:43.7362105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7362228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7362508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7362642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7362816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:33:43.7363087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7363200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7363480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7363619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7363834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:33:43.7364108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7364230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7364509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7364641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7364847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:33:43.7365134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7365432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7365721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-11-23T01:33:43.7365887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:33:43.7366049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:33:43.7366211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:33:43.7366314Z skip: Need at least 2 CUDA devices (2.171s)
2022-11-23T01:33:43.7366321Z 
2022-11-23T01:33:43.7366516Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7366591Z Ran 1 test in 2.171s
2022-11-23T01:33:43.7366596Z 
2022-11-23T01:33:43.7366654Z OK (skipped=1)
2022-11-23T01:33:43.7366659Z 
2022-11-23T01:33:43.7366743Z Generating XML reports...
2022-11-23T01:33:43.7367090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013337.xml
2022-11-23T01:33:43.7367321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7367595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7367801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7368088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7368227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7368231Z 
2022-11-23T01:33:43.7368289Z Running tests...
2022-11-23T01:33:43.7368481Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7368829Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'ucc', 'nccl'} (0.001s)
2022-11-23T01:33:43.7368834Z 
2022-11-23T01:33:43.7369023Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7369098Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7369103Z 
2022-11-23T01:33:43.7369172Z OK (skipped=1)
2022-11-23T01:33:43.7369177Z 
2022-11-23T01:33:43.7369258Z Generating XML reports...
2022-11-23T01:33:43.7369601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013341.xml
2022-11-23T01:33:43.7369832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:33:43.7370095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:33:43.7370218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:33:43.7370498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:33:43.7370666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:33:43.7370671Z 
2022-11-23T01:33:43.7370743Z Running tests...
2022-11-23T01:33:43.7370933Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7371282Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'ucc', 'nccl', 'gloo'} (0.001s)
2022-11-23T01:33:43.7371288Z 
2022-11-23T01:33:43.7371480Z ----------------------------------------------------------------------
2022-11-23T01:33:43.7371582Z Ran 1 test in 0.002s
2022-11-23T01:33:43.7371587Z 
2022-11-23T01:33:43.7371644Z OK (skipped=1)
2022-11-23T01:33:43.7371661Z 
2022-11-23T01:33:43.7371731Z Generating XML reports...
2022-11-23T01:33:43.7372069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml
2022-11-23T01:33:43.7372074Z 
2022-11-23T01:33:43.7372449Z ##[endgroup]
2022-11-23T01:33:43.7372805Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_1yy7fq2y)
2022-11-23T01:33:43.7372810Z 
2022-11-23T01:33:43.7373014Z Running distributed/pipeline/sync/test_worker ... [2022-11-23 01:33:43.480706]
2022-11-23T01:33:43.7373310Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-11-23 01:33:43.480984]
2022-11-23T01:33:45.6294804Z 
2022-11-23T01:33:45.6295466Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker
2022-11-23T01:33:45.6296694Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_hlb3d7iy)
2022-11-23T01:33:45.6297209Z ============================= test session starts ==============================
2022-11-23T01:33:45.6297687Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:45.6297946Z cachedir: .pytest_cache
2022-11-23T01:33:45.6298379Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:45.6298688Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:45.6298929Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:45.6299472Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:45.6299760Z collecting ... collected 6 items
2022-11-23T01:33:45.6300492Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device
2022-11-23T01:33:45.6301074Z 
2022-11-23T01:33:45.6301238Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2022-11-23T01:33:45.6301598Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2022-11-23T01:33:45.6301947Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2022-11-23T01:33:45.6302256Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2022-11-23T01:33:45.6302705Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2022-11-23T01:33:45.6303029Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2022-11-23T01:33:45.6303206Z 
2022-11-23T01:33:45.6303304Z ============================== 6 passed in 0.04s ===============================
2022-11-23T01:33:45.6303440Z 
2022-11-23T01:33:45.6303780Z ##[endgroup]
2022-11-23T01:33:45.6304475Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_hlb3d7iy)
2022-11-23T01:33:45.6304752Z 
2022-11-23T01:33:45.6304971Z Running distributed/pipeline/sync/test_stream ... [2022-11-23 01:33:45.629860]
2022-11-23T01:33:45.6305414Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-11-23 01:33:45.630199]
2022-11-23T01:33:47.3190537Z 
2022-11-23T01:33:47.3191140Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream
2022-11-23T01:33:47.3192730Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_ba_qsarq)
2022-11-23T01:33:47.3193489Z ============================= test session starts ==============================
2022-11-23T01:33:47.3194288Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:47.3194725Z cachedir: .pytest_cache
2022-11-23T01:33:47.3195239Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:47.3195569Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:47.3195880Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:47.3196381Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:47.3196698Z collecting ... collected 19 items
2022-11-23T01:33:47.3198887Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view
2022-11-23T01:33:47.3200775Z 
2022-11-23T01:33:47.3200950Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-11-23T01:33:47.3201299Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%]
2022-11-23T01:33:47.3201669Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-11-23T01:33:47.3202050Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%]
2022-11-23T01:33:47.3202428Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-11-23T01:33:47.3202791Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%]
2022-11-23T01:33:47.3203263Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-11-23T01:33:47.3203620Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%]
2022-11-23T01:33:47.3203959Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-11-23T01:33:47.3204310Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%]
2022-11-23T01:33:47.3204662Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-11-23T01:33:47.3205059Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%]
2022-11-23T01:33:47.3205406Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-11-23T01:33:47.3205777Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%]
2022-11-23T01:33:47.3206146Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%]
2022-11-23T01:33:47.3206513Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%]
2022-11-23T01:33:47.3206874Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-11-23T01:33:47.3207243Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%]
2022-11-23T01:33:47.3207627Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%]
2022-11-23T01:33:47.3207904Z 
2022-11-23T01:33:47.3208024Z ======================== 8 passed, 11 skipped in 0.05s =========================
2022-11-23T01:33:47.3208154Z 
2022-11-23T01:33:47.3208405Z ##[endgroup]
2022-11-23T01:33:47.3208896Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_ba_qsarq)
2022-11-23T01:33:47.3209176Z 
2022-11-23T01:33:47.3209395Z Running distributed/pipeline/sync/test_pipe ... [2022-11-23 01:33:47.319423]
2022-11-23T01:33:47.3209848Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-11-23 01:33:47.319764]
2022-11-23T01:33:50.5310967Z 
2022-11-23T01:33:50.5311550Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2022-11-23T01:33:50.5312737Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_saiim5dg)
2022-11-23T01:33:50.5313392Z ============================= test session starts ==============================
2022-11-23T01:33:50.5313861Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:50.5314105Z cachedir: .pytest_cache
2022-11-23T01:33:50.5314536Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:50.5314864Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:50.5315104Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:50.5315521Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:50.5315818Z collecting ... collected 56 items
2022-11-23T01:33:50.5323648Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2022-11-23T01:33:50.5328144Z 
2022-11-23T01:33:50.5328293Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-11-23T01:33:50.5328614Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-11-23T01:33:50.5328926Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-11-23T01:33:50.5329239Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-11-23T01:33:50.5329539Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-11-23T01:33:50.5329893Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-11-23T01:33:50.5330215Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-11-23T01:33:50.5330509Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-11-23T01:33:50.5330839Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-11-23T01:33:50.5331178Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-11-23T01:33:50.5331509Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-11-23T01:33:50.5331862Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-11-23T01:33:50.5332180Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-11-23T01:33:50.5332480Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-11-23T01:33:50.5332794Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-11-23T01:33:50.5333116Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-11-23T01:33:50.5333428Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-11-23T01:33:50.5333739Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-11-23T01:33:50.5334044Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-11-23T01:33:50.5334348Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-11-23T01:33:50.5334649Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-11-23T01:33:50.5335104Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-11-23T01:33:50.5335430Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-11-23T01:33:50.5335756Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-11-23T01:33:50.5336092Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-11-23T01:33:50.5336452Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-11-23T01:33:50.5336765Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-11-23T01:33:50.5337098Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-11-23T01:33:50.5337431Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-11-23T01:33:50.5337748Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-11-23T01:33:50.5338084Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-11-23T01:33:50.5338408Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-11-23T01:33:50.5338719Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-11-23T01:33:50.5339024Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-11-23T01:33:50.5339349Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-11-23T01:33:50.5339685Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-11-23T01:33:50.5340015Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-11-23T01:33:50.5340364Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-11-23T01:33:50.5340711Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-11-23T01:33:50.5341042Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-11-23T01:33:50.5341335Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-11-23T01:33:50.5341645Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED   [ 75%]
2022-11-23T01:33:50.5342018Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-11-23T01:33:50.5342313Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-11-23T01:33:50.5342615Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-11-23T01:33:50.5342939Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-11-23T01:33:50.5343283Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-11-23T01:33:50.5343624Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%]
2022-11-23T01:33:50.5344009Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%]
2022-11-23T01:33:50.5344366Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-11-23T01:33:50.5344708Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-11-23T01:33:50.5345034Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%]
2022-11-23T01:33:50.5345366Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%]
2022-11-23T01:33:50.5345704Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%]
2022-11-23T01:33:50.5346022Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%]
2022-11-23T01:33:50.5346341Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%]
2022-11-23T01:33:50.5346520Z 
2022-11-23T01:33:50.5346636Z ======================== 48 passed, 8 skipped in 1.55s =========================
2022-11-23T01:33:50.5346777Z 
2022-11-23T01:33:50.5347029Z ##[endgroup]
2022-11-23T01:33:50.5347544Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_saiim5dg)
2022-11-23T01:33:50.5347819Z 
2022-11-23T01:33:50.5348035Z Running distributed/pipeline/sync/test_microbatch ... [2022-11-23 01:33:50.531518]
2022-11-23T01:33:50.5348500Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-11-23 01:33:50.531903]
2022-11-23T01:33:52.2107888Z 
2022-11-23T01:33:52.2108359Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch
2022-11-23T01:33:52.2109328Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_g1t2kf6q)
2022-11-23T01:33:52.2109831Z ============================= test session starts ==============================
2022-11-23T01:33:52.2110351Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:52.2110598Z cachedir: .pytest_cache
2022-11-23T01:33:52.2111090Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:52.2111421Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:52.2111667Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:52.2112132Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:52.2112591Z collecting ... collected 10 items
2022-11-23T01:33:52.2113750Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors
2022-11-23T01:33:52.2114591Z 
2022-11-23T01:33:52.2114751Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2022-11-23T01:33:52.2115094Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2022-11-23T01:33:52.2115526Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2022-11-23T01:33:52.2115844Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2022-11-23T01:33:52.2116253Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2022-11-23T01:33:52.2116580Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2022-11-23T01:33:52.2116884Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2022-11-23T01:33:52.2117205Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2022-11-23T01:33:52.2117532Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2022-11-23T01:33:52.2117872Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2022-11-23T01:33:52.2118049Z 
2022-11-23T01:33:52.2118162Z ============================== 10 passed in 0.04s ==============================
2022-11-23T01:33:52.2118297Z 
2022-11-23T01:33:52.2118541Z ##[endgroup]
2022-11-23T01:33:52.2119049Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_g1t2kf6q)
2022-11-23T01:33:52.2119341Z 
2022-11-23T01:33:52.2119552Z Running distributed/pipeline/sync/test_dependency ... [2022-11-23 01:33:52.211110]
2022-11-23T01:33:52.2120021Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-11-23 01:33:52.211421]
2022-11-23T01:33:53.8687270Z 
2022-11-23T01:33:53.8687942Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency
2022-11-23T01:33:53.8688835Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_tuf10wu1)
2022-11-23T01:33:53.8689306Z ============================= test session starts ==============================
2022-11-23T01:33:53.8689781Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:53.8690084Z cachedir: .pytest_cache
2022-11-23T01:33:53.8690628Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:53.8691057Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:53.8691296Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:53.8691779Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:53.8692082Z collecting ... collected 6 items
2022-11-23T01:33:53.8692828Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad
2022-11-23T01:33:53.8693539Z 
2022-11-23T01:33:53.8693725Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED     [ 16%]
2022-11-23T01:33:53.8694068Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-11-23T01:33:53.8694408Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-11-23T01:33:53.8695085Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-11-23T01:33:53.8695476Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-11-23T01:33:53.8695841Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-11-23T01:33:53.8696036Z 
2022-11-23T01:33:53.8696142Z ========================= 5 passed, 1 skipped in 0.03s =========================
2022-11-23T01:33:53.8696289Z 
2022-11-23T01:33:53.8696543Z ##[endgroup]
2022-11-23T01:33:53.8697053Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_tuf10wu1)
2022-11-23T01:33:53.8697417Z 
2022-11-23T01:33:53.8697631Z Running distributed/pipeline/sync/test_copy ... [2022-11-23 01:33:53.868978]
2022-11-23T01:33:53.8698071Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-11-23 01:33:53.869284]
2022-11-23T01:33:55.5195953Z 
2022-11-23T01:33:55.5196591Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2022-11-23T01:33:55.5197865Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_oq5kwx3y)
2022-11-23T01:33:55.5198460Z ============================= test session starts ==============================
2022-11-23T01:33:55.5198910Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:55.5199170Z cachedir: .pytest_cache
2022-11-23T01:33:55.5199613Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:55.5199991Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:55.5200405Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:55.5201006Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:55.5201303Z collecting ... collected 5 items
2022-11-23T01:33:55.5201873Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2022-11-23T01:33:55.5202334Z 
2022-11-23T01:33:55.5202489Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-11-23T01:33:55.5202813Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED  [ 40%]
2022-11-23T01:33:55.5203138Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED  [ 60%]
2022-11-23T01:33:55.5203694Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%]
2022-11-23T01:33:55.5204180Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-11-23T01:33:55.5204362Z 
2022-11-23T01:33:55.5204478Z ========================= 2 passed, 3 skipped in 0.03s =========================
2022-11-23T01:33:55.5204621Z 
2022-11-23T01:33:55.5204842Z ##[endgroup]
2022-11-23T01:33:55.5205304Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_oq5kwx3y)
2022-11-23T01:33:55.5205581Z 
2022-11-23T01:33:55.5205796Z Running distributed/pipeline/sync/test_bugs ... [2022-11-23 01:33:55.519869]
2022-11-23T01:33:55.5206249Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-11-23 01:33:55.520191]
2022-11-23T01:33:57.3218772Z 
2022-11-23T01:33:57.3219380Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs
2022-11-23T01:33:57.3220947Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_uoyafxl6)
2022-11-23T01:33:57.3221519Z ============================= test session starts ==============================
2022-11-23T01:33:57.3222034Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:57.3222296Z cachedir: .pytest_cache
2022-11-23T01:33:57.3222724Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:57.3223159Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:57.3223586Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:57.3224198Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:57.3224587Z collecting ... collected 4 items
2022-11-23T01:33:57.3225139Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms
2022-11-23T01:33:57.3225579Z 
2022-11-23T01:33:57.3225772Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2022-11-23T01:33:57.3226355Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2022-11-23T01:33:57.3226854Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%]
2022-11-23T01:33:57.3227158Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2022-11-23T01:33:57.3227336Z 
2022-11-23T01:33:57.3227453Z ========================= 3 passed, 1 skipped in 0.17s =========================
2022-11-23T01:33:57.3227596Z 
2022-11-23T01:33:57.3227831Z ##[endgroup]
2022-11-23T01:33:57.3228299Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_uoyafxl6)
2022-11-23T01:33:57.3228574Z 
2022-11-23T01:33:57.3228816Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-11-23 01:33:57.322132]
2022-11-23T01:33:57.3229318Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-11-23 01:33:57.322449]
2022-11-23T01:33:58.9835421Z 
2022-11-23T01:33:58.9836010Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:33:58.9837347Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_5xh5t3up)
2022-11-23T01:33:58.9838187Z ============================= test session starts ==============================
2022-11-23T01:33:58.9838750Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:33:58.9839006Z cachedir: .pytest_cache
2022-11-23T01:33:58.9839485Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:33:58.9839834Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:33:58.9840072Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:33:58.9840555Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:33:58.9840838Z collecting ... collected 9 items
2022-11-23T01:33:58.9842074Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated
2022-11-23T01:33:58.9843131Z 
2022-11-23T01:33:58.9843286Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-11-23T01:33:58.9843692Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-11-23T01:33:58.9844050Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-11-23T01:33:58.9844517Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-11-23T01:33:58.9844850Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-11-23T01:33:58.9845270Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-11-23T01:33:58.9845667Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-11-23T01:33:58.9846072Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-11-23T01:33:58.9846452Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-11-23T01:33:58.9846712Z 
2022-11-23T01:33:58.9846823Z ============================== 9 passed in 0.03s ===============================
2022-11-23T01:33:58.9846957Z 
2022-11-23T01:33:58.9847190Z ##[endgroup]
2022-11-23T01:33:58.9847862Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_5xh5t3up)
2022-11-23T01:33:58.9848236Z 
2022-11-23T01:33:58.9848469Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-11-23 01:33:58.983807]
2022-11-23T01:33:58.9849001Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-11-23 01:33:58.984130]
2022-11-23T01:34:00.6473235Z 
2022-11-23T01:34:00.6473810Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2022-11-23T01:34:00.6474807Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_c72ziyla)
2022-11-23T01:34:00.6475281Z ============================= test session starts ==============================
2022-11-23T01:34:00.6475836Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:34:00.6476081Z cachedir: .pytest_cache
2022-11-23T01:34:00.6476561Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:34:00.6476885Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:34:00.6477131Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:34:00.6477602Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:34:00.6477925Z collecting ... collected 7 items
2022-11-23T01:34:00.6478729Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2022-11-23T01:34:00.6479379Z 
2022-11-23T01:34:00.6479550Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-11-23T01:34:00.6480071Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-11-23T01:34:00.6480448Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-11-23T01:34:00.6480804Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-11-23T01:34:00.6481196Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-11-23T01:34:00.6481521Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-11-23T01:34:00.6481894Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-11-23T01:34:00.6482137Z 
2022-11-23T01:34:00.6482247Z ============================== 7 passed in 0.03s ===============================
2022-11-23T01:34:00.6482383Z 
2022-11-23T01:34:00.6482660Z ##[endgroup]
2022-11-23T01:34:00.6483180Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_c72ziyla)
2022-11-23T01:34:00.6483531Z 
2022-11-23T01:34:00.6483743Z Running distributed/pipeline/sync/skip/test_leak ... [2022-11-23 01:34:00.647654]
2022-11-23T01:34:00.6484265Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-11-23 01:34:00.647979]
2022-11-23T01:34:02.4843937Z 
2022-11-23T01:34:02.4844571Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak
2022-11-23T01:34:02.4845796Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_nwjq2lrb)
2022-11-23T01:34:02.4846286Z ============================= test session starts ==============================
2022-11-23T01:34:02.4846747Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:34:02.4846996Z cachedir: .pytest_cache
2022-11-23T01:34:02.4847432Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:34:02.4847854Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:34:02.4848093Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:34:02.4848509Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:34:02.4848805Z collecting ... collected 8 items
2022-11-23T01:34:02.4850255Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval]
2022-11-23T01:34:02.4851099Z 
2022-11-23T01:34:02.4851357Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2022-11-23T01:34:02.4851800Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2022-11-23T01:34:02.4852280Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2022-11-23T01:34:02.4852740Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2022-11-23T01:34:02.4853183Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2022-11-23T01:34:02.4853616Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2022-11-23T01:34:02.4854145Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2022-11-23T01:34:02.4854506Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2022-11-23T01:34:02.4854696Z 
2022-11-23T01:34:02.4854805Z ============================== 8 passed in 0.20s ===============================
2022-11-23T01:34:02.4855126Z 
2022-11-23T01:34:02.4855372Z ##[endgroup]
2022-11-23T01:34:02.4855870Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_nwjq2lrb)
2022-11-23T01:34:02.4856238Z 
2022-11-23T01:34:02.4856463Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-11-23 01:34:02.484704]
2022-11-23T01:34:02.4856929Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-11-23 01:34:02.485048]
2022-11-23T01:34:04.2124295Z 
2022-11-23T01:34:04.2124870Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:34:04.2126134Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_apgyri69)
2022-11-23T01:34:04.2126917Z ============================= test session starts ==============================
2022-11-23T01:34:04.2127799Z platform linux -- Python 3.7.13, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T01:34:04.2128110Z cachedir: .pytest_cache
2022-11-23T01:34:04.2128595Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T01:34:04.2128958Z torch: 1.14.0a0+git1cfd385
2022-11-23T01:34:04.2129191Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T01:34:04.2129621Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T01:34:04.2129923Z collecting ... collected 13 items
2022-11-23T01:34:04.2131713Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip
2022-11-23T01:34:04.2132813Z 
2022-11-23T01:34:04.2133044Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [  7%]
2022-11-23T01:34:04.2133444Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%]
2022-11-23T01:34:04.2133825Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%]
2022-11-23T01:34:04.2134224Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-11-23T01:34:04.2134618Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%]
2022-11-23T01:34:04.2135159Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%]
2022-11-23T01:34:04.2135564Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%]
2022-11-23T01:34:04.2135962Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-11-23T01:34:04.2136548Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%]
2022-11-23T01:34:04.2136946Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%]
2022-11-23T01:34:04.2137355Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%]
2022-11-23T01:34:04.2137763Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-11-23T01:34:04.2138082Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-11-23T01:34:04.2138262Z 
2022-11-23T01:34:04.2138380Z ======================== 1 passed, 12 skipped in 0.07s =========================
2022-11-23T01:34:04.2138587Z 
2022-11-23T01:34:04.2138845Z ##[endgroup]
2022-11-23T01:34:04.2139335Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_apgyri69)
2022-11-23T01:34:04.2139630Z 
2022-11-23T01:34:04.2139869Z Running distributed/optim/test_apply_optimizer_in_backward ... [2022-11-23 01:34:04.212726]
2022-11-23T01:34:04.2140425Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:04.213105]
2022-11-23T01:34:05.6439408Z 
2022-11-23T01:34:05.6440046Z Expand the folded group to see the log file of distributed/optim/test_apply_optimizer_in_backward
2022-11-23T01:34:05.6441044Z ##[group]PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_xd0aflp0)
2022-11-23T01:34:05.6441366Z 
2022-11-23T01:34:05.6441567Z ##[endgroup]
2022-11-23T01:34:05.6442165Z FINISHED PRINTING LOG FILE of distributed/optim/test_apply_optimizer_in_backward (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_apply_optimizer_in_backward_xd0aflp0)
2022-11-23T01:34:05.6442525Z 
2022-11-23T01:34:05.6443620Z Running distributed/elastic/timer/api_test ... [2022-11-23 01:34:05.644162]
2022-11-23T01:34:05.6447030Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:05.644502]
2022-11-23T01:34:06.9085779Z 
2022-11-23T01:34:06.9086335Z Expand the folded group to see the log file of distributed/elastic/timer/api_test
2022-11-23T01:34:06.9087679Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_oa3pejmb)
2022-11-23T01:34:06.9088188Z 
2022-11-23T01:34:06.9093165Z ##[endgroup]
2022-11-23T01:34:06.9093939Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_oa3pejmb)
2022-11-23T01:34:06.9094461Z 
2022-11-23T01:34:06.9097928Z Running distributed/checkpoint/test_traverse ... [2022-11-23 01:34:06.909543]
2022-11-23T01:34:06.9101087Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:06.909917]
2022-11-23T01:34:08.7303012Z 
2022-11-23T01:34:08.7303754Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse
2022-11-23T01:34:08.7305015Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse__3iv9hi1)
2022-11-23T01:34:08.7306154Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse
2022-11-23T01:34:08.7306474Z 
2022-11-23T01:34:08.7306605Z Running tests...
2022-11-23T01:34:08.7307128Z ----------------------------------------------------------------------
2022-11-23T01:34:08.7307600Z   test_get_element (__main__.TestTraverse) ... ok (0.244s)
2022-11-23T01:34:08.7307926Z   test_set_element (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T01:34:08.7308422Z   test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T01:34:08.7308794Z   test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T01:34:08.7309071Z   test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T01:34:08.7309393Z   test_traverse_shallow (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T01:34:08.7309660Z   test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T01:34:08.7309820Z 
2022-11-23T01:34:08.7310045Z ----------------------------------------------------------------------
2022-11-23T01:34:08.7310307Z Ran 7 tests in 0.254s
2022-11-23T01:34:08.7310480Z 
2022-11-23T01:34:08.7310549Z OK
2022-11-23T01:34:08.7310627Z 
2022-11-23T01:34:08.7310711Z Generating XML reports...
2022-11-23T01:34:08.7311223Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123013408.xml
2022-11-23T01:34:08.7311480Z 
2022-11-23T01:34:08.7311797Z ##[endgroup]
2022-11-23T01:34:08.7312418Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse__3iv9hi1)
2022-11-23T01:34:08.7312699Z 
2022-11-23T01:34:08.7312897Z Running distributed/_shard/test_sharder ... [2022-11-23 01:34:08.730647]
2022-11-23T01:34:08.7313460Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:08.731061]
2022-11-23T01:34:10.1535840Z 
2022-11-23T01:34:10.1536506Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2022-11-23T01:34:10.1537501Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_wag2o990)
2022-11-23T01:34:10.1537963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppg5w97b
2022-11-23T01:34:10.1538384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppg5w97b/_remote_module_non_scriptable.py
2022-11-23T01:34:10.1538607Z 
2022-11-23T01:34:10.1538808Z ##[endgroup]
2022-11-23T01:34:10.1539389Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_wag2o990)
2022-11-23T01:34:10.1539685Z 
2022-11-23T01:34:10.1540033Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-11-23 01:34:10.153827]
2022-11-23T01:34:10.1543210Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:10.154161]
2022-11-23T01:34:11.5744819Z 
2022-11-23T01:34:11.5745477Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2022-11-23T01:34:11.5746318Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_b4yv64k0)
2022-11-23T01:34:11.5746668Z 
2022-11-23T01:34:11.5746874Z ##[endgroup]
2022-11-23T01:34:11.5747568Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_b4yv64k0)
2022-11-23T01:34:11.5747875Z 
2022-11-23T01:34:11.5749168Z Running distributed/test_c10d_spawn_ucc ... [2022-11-23 01:34:11.574718]
2022-11-23T01:34:11.5752323Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:11.575075]
2022-11-23T01:34:28.9093608Z 
2022-11-23T01:34:28.9094232Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc
2022-11-23T01:34:28.9095573Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_a9aqw5du)
2022-11-23T01:34:28.9104683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo3431io
2022-11-23T01:34:28.9105124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo3431io/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9105800Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_ucc>]>
2022-11-23T01:34:28.9106397Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:34:28.9106712Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:34:28.9107159Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:34:28.9107428Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:34:28.9107681Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:34:28.9108463Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsUcc testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]>
2022-11-23T01:34:28.9109209Z test_all_gather (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9109536Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9109817Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9110173Z test_allreduce (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9110462Z test_broadcast (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9110753Z test_reduce (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T01:34:28.9111324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9111717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9112155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9112550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9112897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18eo3pvz
2022-11-23T01:34:28.9113363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18eo3pvz/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9113808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9114067Z 
2022-11-23T01:34:28.9114142Z Running tests...
2022-11-23T01:34:28.9114445Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9114862Z   test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:34:28.9115084Z 
2022-11-23T01:34:28.9115266Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9115562Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9115676Z 
2022-11-23T01:34:28.9115748Z OK (skipped=1)
2022-11-23T01:34:28.9115854Z 
2022-11-23T01:34:28.9115925Z Generating XML reports...
2022-11-23T01:34:28.9116460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013414.xml
2022-11-23T01:34:28.9117071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9117408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9117885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9118234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9118685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuho4zp7j
2022-11-23T01:34:28.9119196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuho4zp7j/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9119713Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9119913Z 
2022-11-23T01:34:28.9119986Z Running tests...
2022-11-23T01:34:28.9120348Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9120697Z   test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:34:28.9121018Z 
2022-11-23T01:34:28.9121215Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9121455Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9121568Z 
2022-11-23T01:34:28.9121665Z OK (skipped=1)
2022-11-23T01:34:28.9121789Z 
2022-11-23T01:34:28.9121872Z Generating XML reports...
2022-11-23T01:34:28.9122356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013416.xml
2022-11-23T01:34:28.9122968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9123342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9123771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9124174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9124524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi65zoepg
2022-11-23T01:34:28.9124969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi65zoepg/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9125424Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9125682Z 
2022-11-23T01:34:28.9125759Z Running tests...
2022-11-23T01:34:28.9126048Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9126471Z   test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:34:28.9126690Z 
2022-11-23T01:34:28.9126885Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9127175Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9127289Z 
2022-11-23T01:34:28.9127350Z OK (skipped=1)
2022-11-23T01:34:28.9127457Z 
2022-11-23T01:34:28.9127610Z Generating XML reports...
2022-11-23T01:34:28.9128156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013417.xml
2022-11-23T01:34:28.9128772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9129092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9129588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9129936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9130327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3iityj1
2022-11-23T01:34:28.9130731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3iityj1/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9131226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9131435Z 
2022-11-23T01:34:28.9131509Z Running tests...
2022-11-23T01:34:28.9131867Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9132219Z   test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9132440Z 
2022-11-23T01:34:28.9132744Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9132982Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9133093Z 
2022-11-23T01:34:28.9133151Z OK (skipped=1)
2022-11-23T01:34:28.9133257Z 
2022-11-23T01:34:28.9133341Z Generating XML reports...
2022-11-23T01:34:28.9133826Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013419.xml
2022-11-23T01:34:28.9134381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9134782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9135326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9135671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9136022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hm1uq8l
2022-11-23T01:34:28.9136409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hm1uq8l/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9136862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9137061Z 
2022-11-23T01:34:28.9137133Z Running tests...
2022-11-23T01:34:28.9137420Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9137782Z   test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9138004Z 
2022-11-23T01:34:28.9138196Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9138435Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9138593Z 
2022-11-23T01:34:28.9138670Z OK (skipped=1)
2022-11-23T01:34:28.9138775Z 
2022-11-23T01:34:28.9138859Z Generating XML reports...
2022-11-23T01:34:28.9139341Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013421.xml
2022-11-23T01:34:28.9139882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9140209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9140637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9140985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9141316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27gtluhm
2022-11-23T01:34:28.9141722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27gtluhm/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9142173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9142368Z 
2022-11-23T01:34:28.9142441Z Running tests...
2022-11-23T01:34:28.9142729Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9143102Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9143329Z 
2022-11-23T01:34:28.9143522Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9143745Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9143857Z 
2022-11-23T01:34:28.9143928Z OK (skipped=1)
2022-11-23T01:34:28.9144034Z 
2022-11-23T01:34:28.9144118Z Generating XML reports...
2022-11-23T01:34:28.9144599Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013423.xml
2022-11-23T01:34:28.9145133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9145474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9146033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9146374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9146718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5zng03u
2022-11-23T01:34:28.9147121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5zng03u/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9147572Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9147808Z 
2022-11-23T01:34:28.9147867Z Running tests...
2022-11-23T01:34:28.9148164Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9148526Z   test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9148748Z 
2022-11-23T01:34:28.9148941Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9149165Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9149276Z 
2022-11-23T01:34:28.9149349Z OK (skipped=1)
2022-11-23T01:34:28.9149455Z 
2022-11-23T01:34:28.9149539Z Generating XML reports...
2022-11-23T01:34:28.9149998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013424.xml
2022-11-23T01:34:28.9150545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9150877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9151308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9151641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9151982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprj80nhvf
2022-11-23T01:34:28.9152386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprj80nhvf/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9152824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9153019Z 
2022-11-23T01:34:28.9153092Z Running tests...
2022-11-23T01:34:28.9153391Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9153758Z   test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9153978Z 
2022-11-23T01:34:28.9154160Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9154395Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9154506Z 
2022-11-23T01:34:28.9154576Z OK (skipped=1)
2022-11-23T01:34:28.9154684Z 
2022-11-23T01:34:28.9154755Z Generating XML reports...
2022-11-23T01:34:28.9155235Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013426.xml
2022-11-23T01:34:28.9155787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:34:28.9156116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:34:28.9156531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:34:28.9156875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:34:28.9157221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_4uh8vm
2022-11-23T01:34:28.9157619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_4uh8vm/_remote_module_non_scriptable.py
2022-11-23T01:34:28.9158056Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T01:34:28.9158250Z 
2022-11-23T01:34:28.9158321Z Running tests...
2022-11-23T01:34:28.9158655Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9159072Z   test_reduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2022-11-23T01:34:28.9159294Z 
2022-11-23T01:34:28.9159490Z ----------------------------------------------------------------------
2022-11-23T01:34:28.9159727Z Ran 1 test in 0.001s
2022-11-23T01:34:28.9159839Z 
2022-11-23T01:34:28.9159910Z OK (skipped=1)
2022-11-23T01:34:28.9160003Z 
2022-11-23T01:34:28.9160087Z Generating XML reports...
2022-11-23T01:34:28.9160561Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013428.xml
2022-11-23T01:34:28.9160879Z 
2022-11-23T01:34:28.9161196Z ##[endgroup]
2022-11-23T01:34:28.9161620Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_a9aqw5du)
2022-11-23T01:34:28.9161867Z 
2022-11-23T01:34:28.9162095Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-11-23 01:34:28.909830]
2022-11-23T01:34:28.9162652Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:28.910163]
2022-11-23T01:34:30.3532192Z 
2022-11-23T01:34:30.3532843Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_softmax
2022-11-23T01:34:30.3533961Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_x3bgn9go)
2022-11-23T01:34:30.3534491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpxtnjqx
2022-11-23T01:34:30.3535042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpxtnjqx/_remote_module_non_scriptable.py
2022-11-23T01:34:30.3535639Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2022-11-23T01:34:30.3535923Z 
2022-11-23T01:34:30.3535984Z Running tests...
2022-11-23T01:34:30.3536300Z ----------------------------------------------------------------------
2022-11-23T01:34:30.3536750Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:30.3537451Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:30.3537836Z 
2022-11-23T01:34:30.3538040Z ----------------------------------------------------------------------
2022-11-23T01:34:30.3538285Z Ran 2 tests in 0.001s
2022-11-23T01:34:30.3538398Z 
2022-11-23T01:34:30.3538472Z OK (skipped=2)
2022-11-23T01:34:30.3538579Z 
2022-11-23T01:34:30.3538653Z Generating XML reports...
2022-11-23T01:34:30.3539190Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221123013430.xml
2022-11-23T01:34:30.3539469Z 
2022-11-23T01:34:30.3539743Z ##[endgroup]
2022-11-23T01:34:30.3540401Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_x3bgn9go)
2022-11-23T01:34:30.3540957Z 
2022-11-23T01:34:30.3541248Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-11-23 01:34:30.353537]
2022-11-23T01:34:30.3541825Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:30.353877]
2022-11-23T01:34:31.7917532Z 
2022-11-23T01:34:31.7918168Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-11-23T01:34:31.7919787Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_h3handrs)
2022-11-23T01:34:31.7920494Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-11-23T01:34:31.7920726Z 
2022-11-23T01:34:31.7920816Z Running tests...
2022-11-23T01:34:31.7921194Z ----------------------------------------------------------------------
2022-11-23T01:34:31.7921578Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:31.7922233Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:31.7922740Z 
2022-11-23T01:34:31.7922946Z ----------------------------------------------------------------------
2022-11-23T01:34:31.7923176Z Ran 2 tests in 0.001s
2022-11-23T01:34:31.7923290Z 
2022-11-23T01:34:31.7923364Z OK (skipped=2)
2022-11-23T01:34:31.7923473Z 
2022-11-23T01:34:31.7923561Z Generating XML reports...
2022-11-23T01:34:31.7924083Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123013431.xml
2022-11-23T01:34:31.7924432Z 
2022-11-23T01:34:31.7924665Z ##[endgroup]
2022-11-23T01:34:31.7925613Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_h3handrs)
2022-11-23T01:34:31.7925920Z 
2022-11-23T01:34:31.7926158Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-11-23 01:34:31.792029]
2022-11-23T01:34:31.7926709Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:31.792357]
2022-11-23T01:34:33.2272805Z 
2022-11-23T01:34:33.2273376Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-11-23T01:34:33.2274530Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_yd999y79)
2022-11-23T01:34:33.2275193Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-11-23T01:34:33.2275421Z 
2022-11-23T01:34:33.2275497Z Running tests...
2022-11-23T01:34:33.2275807Z ----------------------------------------------------------------------
2022-11-23T01:34:33.2276116Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-11-23T01:34:33.2276464Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:34:33.2277009Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:33.2277499Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-11-23T01:34:33.2278112Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:33.2278532Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:34:33.2278760Z 
2022-11-23T01:34:33.2278964Z ----------------------------------------------------------------------
2022-11-23T01:34:33.2279203Z Ran 4 tests in 0.002s
2022-11-23T01:34:33.2279315Z 
2022-11-23T01:34:33.2279375Z OK (skipped=4)
2022-11-23T01:34:33.2279481Z 
2022-11-23T01:34:33.2279568Z Generating XML reports...
2022-11-23T01:34:33.2280273Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123013432.xml
2022-11-23T01:34:33.2280704Z 
2022-11-23T01:34:33.2281103Z ##[endgroup]
2022-11-23T01:34:33.2281939Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_yd999y79)
2022-11-23T01:34:33.2282238Z 
2022-11-23T01:34:33.2282490Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-11-23 01:34:33.227561]
2022-11-23T01:34:33.2283073Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:33.227905]
2022-11-23T01:34:34.6618581Z 
2022-11-23T01:34:34.6619320Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T01:34:34.6620507Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_lcm53vv3)
2022-11-23T01:34:34.6621206Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-11-23T01:34:34.6621444Z 
2022-11-23T01:34:34.6621521Z Running tests...
2022-11-23T01:34:34.6621889Z ----------------------------------------------------------------------
2022-11-23T01:34:34.6622280Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-11-23T01:34:34.6622529Z 
2022-11-23T01:34:34.6622729Z ----------------------------------------------------------------------
2022-11-23T01:34:34.6622969Z Ran 1 test in 0.003s
2022-11-23T01:34:34.6623082Z 
2022-11-23T01:34:34.6623155Z OK (skipped=1)
2022-11-23T01:34:34.6623256Z 
2022-11-23T01:34:34.6623340Z Generating XML reports...
2022-11-23T01:34:34.6623888Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123013434.xml
2022-11-23T01:34:34.6624210Z 
2022-11-23T01:34:34.6624430Z ##[endgroup]
2022-11-23T01:34:34.6624958Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_lcm53vv3)
2022-11-23T01:34:34.6625295Z 
2022-11-23T01:34:34.6625524Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-11-23 01:34:34.662137]
2022-11-23T01:34:34.6626284Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:34.662468]
2022-11-23T01:34:36.0898715Z 
2022-11-23T01:34:36.0899242Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T01:34:36.0900317Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_cy4hz0_g)
2022-11-23T01:34:36.0900987Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-11-23T01:34:36.0901205Z 
2022-11-23T01:34:36.0901279Z Running tests...
2022-11-23T01:34:36.0901653Z ----------------------------------------------------------------------
2022-11-23T01:34:36.0902086Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:34:36.0902607Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:34:36.0902864Z 
2022-11-23T01:34:36.0903064Z ----------------------------------------------------------------------
2022-11-23T01:34:36.0903308Z Ran 2 tests in 0.001s
2022-11-23T01:34:36.0903408Z 
2022-11-23T01:34:36.0903479Z OK (skipped=2)
2022-11-23T01:34:36.0903584Z 
2022-11-23T01:34:36.0903667Z Generating XML reports...
2022-11-23T01:34:36.0904174Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123013435.xml
2022-11-23T01:34:36.0904671Z 
2022-11-23T01:34:36.0904952Z ##[endgroup]
2022-11-23T01:34:36.0905561Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_cy4hz0_g)
2022-11-23T01:34:36.0905853Z 
2022-11-23T01:34:36.0906058Z Running distributed/test_c10d_spawn_nccl ... [2022-11-23 01:34:36.090100]
2022-11-23T01:34:36.0906596Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:34:36.090432]
2022-11-23T01:35:00.3460606Z 
2022-11-23T01:35:00.3461119Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2022-11-23T01:35:00.3462288Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_pbcv0das)
2022-11-23T01:35:00.3462961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmml_h_nm
2022-11-23T01:35:00.3464083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmml_h_nm/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3465218Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]>
2022-11-23T01:35:00.3466346Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:35:00.3466812Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:35:00.3467441Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:35:00.3467973Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-11-23T01:35:00.3468232Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:35:00.3468473Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:35:00.3469561Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2022-11-23T01:35:00.3470896Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3471298Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3471580Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3471881Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3472187Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3472482Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3472760Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3473059Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3473380Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T01:35:00.3473932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3474271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3474710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3475061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3475571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29l1t3fe
2022-11-23T01:35:00.3475976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29l1t3fe/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3476439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3476639Z 
2022-11-23T01:35:00.3476699Z Running tests...
2022-11-23T01:35:00.3477007Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3477374Z   test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:35:00.3477659Z 
2022-11-23T01:35:00.3477852Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3478080Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3478191Z 
2022-11-23T01:35:00.3478264Z OK (skipped=1)
2022-11-23T01:35:00.3478371Z 
2022-11-23T01:35:00.3478457Z Generating XML reports...
2022-11-23T01:35:00.3478930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013438.xml
2022-11-23T01:35:00.3479491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3479827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3480259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3480593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3480942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqbukxqp
2022-11-23T01:35:00.3481345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqbukxqp/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3481803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3481987Z 
2022-11-23T01:35:00.3482059Z Running tests...
2022-11-23T01:35:00.3482357Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3482720Z   test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:35:00.3482939Z 
2022-11-23T01:35:00.3483119Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3483357Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3483472Z 
2022-11-23T01:35:00.3483543Z OK (skipped=1)
2022-11-23T01:35:00.3483652Z 
2022-11-23T01:35:00.3483735Z Generating XML reports...
2022-11-23T01:35:00.3484194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013440.xml
2022-11-23T01:35:00.3484741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3485072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3485490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3485836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3486183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewu72a9h
2022-11-23T01:35:00.3486582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewu72a9h/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3487022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3487220Z 
2022-11-23T01:35:00.3487291Z Running tests...
2022-11-23T01:35:00.3487674Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3488024Z   test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:35:00.3488243Z 
2022-11-23T01:35:00.3488480Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3488717Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3488828Z 
2022-11-23T01:35:00.3488900Z OK (skipped=1)
2022-11-23T01:35:00.3489006Z 
2022-11-23T01:35:00.3489077Z Generating XML reports...
2022-11-23T01:35:00.3489550Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013442.xml
2022-11-23T01:35:00.3490104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3490468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3490884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3491231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3491576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmckxbi5
2022-11-23T01:35:00.3491966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmckxbi5/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3492420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3492615Z 
2022-11-23T01:35:00.3492687Z Running tests...
2022-11-23T01:35:00.3492985Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3493330Z   test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-11-23T01:35:00.3493547Z 
2022-11-23T01:35:00.3493741Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3493976Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3494086Z 
2022-11-23T01:35:00.3494143Z OK (skipped=1)
2022-11-23T01:35:00.3494251Z 
2022-11-23T01:35:00.3494336Z Generating XML reports...
2022-11-23T01:35:00.3494806Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013444.xml
2022-11-23T01:35:00.3495558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3495876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3496308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3496654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3496989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqou5x46s
2022-11-23T01:35:00.3497393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqou5x46s/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3497845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3498040Z 
2022-11-23T01:35:00.3498115Z Running tests...
2022-11-23T01:35:00.3498404Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3498779Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3499003Z 
2022-11-23T01:35:00.3499198Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3499436Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3499536Z 
2022-11-23T01:35:00.3499607Z OK (skipped=1)
2022-11-23T01:35:00.3499715Z 
2022-11-23T01:35:00.3499798Z Generating XML reports...
2022-11-23T01:35:00.3500282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013445.xml
2022-11-23T01:35:00.3500827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3501159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3501640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3501985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3502317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphub7jkow
2022-11-23T01:35:00.3502721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphub7jkow/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3503173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3503413Z 
2022-11-23T01:35:00.3503484Z Running tests...
2022-11-23T01:35:00.3503771Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3504146Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:00.3504375Z 
2022-11-23T01:35:00.3504571Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3504795Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3504907Z 
2022-11-23T01:35:00.3504978Z OK (skipped=1)
2022-11-23T01:35:00.3505083Z 
2022-11-23T01:35:00.3505166Z Generating XML reports...
2022-11-23T01:35:00.3505647Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013447.xml
2022-11-23T01:35:00.3506188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3506524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3506948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3507282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3507628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxmgc6ng
2022-11-23T01:35:00.3508028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxmgc6ng/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3508477Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3508659Z 
2022-11-23T01:35:00.3508734Z Running tests...
2022-11-23T01:35:00.3509030Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3509393Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3509618Z 
2022-11-23T01:35:00.3509813Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3510032Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3510143Z 
2022-11-23T01:35:00.3510215Z OK (skipped=1)
2022-11-23T01:35:00.3510321Z 
2022-11-23T01:35:00.3510405Z Generating XML reports...
2022-11-23T01:35:00.3510872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013449.xml
2022-11-23T01:35:00.3511428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3511757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3512183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3512517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3512860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5hrxzkl
2022-11-23T01:35:00.3513264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5hrxzkl/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3513700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3513895Z 
2022-11-23T01:35:00.3513969Z Running tests...
2022-11-23T01:35:00.3514302Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3514682Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3514897Z 
2022-11-23T01:35:00.3515088Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3515323Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3515434Z 
2022-11-23T01:35:00.3515504Z OK (skipped=1)
2022-11-23T01:35:00.3515610Z 
2022-11-23T01:35:00.3515681Z Generating XML reports...
2022-11-23T01:35:00.3516195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013451.xml
2022-11-23T01:35:00.3516745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3517073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3517492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3517834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3518174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20zgmhkc
2022-11-23T01:35:00.3518576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20zgmhkc/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3519011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3519209Z 
2022-11-23T01:35:00.3519287Z Running tests...
2022-11-23T01:35:00.3519589Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3519943Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3520211Z 
2022-11-23T01:35:00.3520406Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3520643Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3520757Z 
2022-11-23T01:35:00.3520828Z OK (skipped=1)
2022-11-23T01:35:00.3520924Z 
2022-11-23T01:35:00.3521007Z Generating XML reports...
2022-11-23T01:35:00.3521488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013452.xml
2022-11-23T01:35:00.3522041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3522359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3522789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3523129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3523470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp530ns3dt
2022-11-23T01:35:00.3523859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp530ns3dt/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3524311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3524507Z 
2022-11-23T01:35:00.3524580Z Running tests...
2022-11-23T01:35:00.3524865Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3525230Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3525452Z 
2022-11-23T01:35:00.3525648Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3525883Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3525995Z 
2022-11-23T01:35:00.3526053Z OK (skipped=1)
2022-11-23T01:35:00.3526158Z 
2022-11-23T01:35:00.3526242Z Generating XML reports...
2022-11-23T01:35:00.3526751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013454.xml
2022-11-23T01:35:00.3527309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3527699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3528131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3528476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3528807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2qpoh9g
2022-11-23T01:35:00.3529250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2qpoh9g/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3529702Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3529903Z 
2022-11-23T01:35:00.3529975Z Running tests...
2022-11-23T01:35:00.3530267Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3530631Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:00.3530852Z 
2022-11-23T01:35:00.3531046Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3531269Z Ran 1 test in 0.000s
2022-11-23T01:35:00.3531384Z 
2022-11-23T01:35:00.3531455Z OK (skipped=1)
2022-11-23T01:35:00.3531560Z 
2022-11-23T01:35:00.3531644Z Generating XML reports...
2022-11-23T01:35:00.3532128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013456.xml
2022-11-23T01:35:00.3532676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3533005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3533435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3533784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3534115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnko0g9w_
2022-11-23T01:35:00.3534517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnko0g9w_/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3535149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3535346Z 
2022-11-23T01:35:00.3535408Z Running tests...
2022-11-23T01:35:00.3535709Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3536083Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:00.3536314Z 
2022-11-23T01:35:00.3536506Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3536734Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3536845Z 
2022-11-23T01:35:00.3536918Z OK (skipped=1)
2022-11-23T01:35:00.3537028Z 
2022-11-23T01:35:00.3537112Z Generating XML reports...
2022-11-23T01:35:00.3537584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013458.xml
2022-11-23T01:35:00.3538138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:00.3538467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:00.3538900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:00.3539233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:00.3539578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjk6ca_g
2022-11-23T01:35:00.3540036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjk6ca_g/_remote_module_non_scriptable.py
2022-11-23T01:35:00.3540496Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T01:35:00.3540682Z 
2022-11-23T01:35:00.3540755Z Running tests...
2022-11-23T01:35:00.3541056Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3541446Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:00.3541693Z 
2022-11-23T01:35:00.3541911Z ----------------------------------------------------------------------
2022-11-23T01:35:00.3542153Z Ran 1 test in 0.001s
2022-11-23T01:35:00.3542262Z 
2022-11-23T01:35:00.3542333Z OK (skipped=1)
2022-11-23T01:35:00.3542440Z 
2022-11-23T01:35:00.3542522Z Generating XML reports...
2022-11-23T01:35:00.3542993Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013459.xml
2022-11-23T01:35:00.3543279Z 
2022-11-23T01:35:00.3543630Z ##[endgroup]
2022-11-23T01:35:00.3544076Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_pbcv0das)
2022-11-23T01:35:00.3544327Z 
2022-11-23T01:35:00.3544544Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-11-23 01:35:00.346521]
2022-11-23T01:35:00.3545123Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:00.346887]
2022-11-23T01:35:01.8019979Z 
2022-11-23T01:35:01.8020741Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T01:35:01.8021814Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_1392e18i)
2022-11-23T01:35:01.8022499Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2022-11-23T01:35:01.8022731Z 
2022-11-23T01:35:01.8022792Z Running tests...
2022-11-23T01:35:01.8023101Z ----------------------------------------------------------------------
2022-11-23T01:35:01.8023443Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T01:35:01.8023992Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:01.8024416Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T01:35:01.8025000Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:01.8025638Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T01:35:01.8026042Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:01.8026437Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T01:35:01.8026840Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:01.8027319Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T01:35:01.8027818Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:01.8028626Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:01.8028874Z 
2022-11-23T01:35:01.8029088Z ----------------------------------------------------------------------
2022-11-23T01:35:01.8029512Z Ran 6 tests in 0.004s
2022-11-23T01:35:01.8029631Z 
2022-11-23T01:35:01.8029690Z OK (skipped=6)
2022-11-23T01:35:01.8029797Z 
2022-11-23T01:35:01.8029882Z Generating XML reports...
2022-11-23T01:35:01.8030449Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123013501.xml
2022-11-23T01:35:01.8030781Z 
2022-11-23T01:35:01.8031019Z ##[endgroup]
2022-11-23T01:35:01.8031543Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_1392e18i)
2022-11-23T01:35:01.8031916Z 
2022-11-23T01:35:01.8032133Z Running distributed/_shard/test_partial_tensor ... [2022-11-23 01:35:01.802318]
2022-11-23T01:35:01.8032671Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:01.802647]
2022-11-23T01:35:03.2348838Z 
2022-11-23T01:35:03.2349311Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2022-11-23T01:35:03.2350423Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_hdew70jd)
2022-11-23T01:35:03.2351041Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2022-11-23T01:35:03.2351253Z 
2022-11-23T01:35:03.2351331Z Running tests...
2022-11-23T01:35:03.2351626Z ----------------------------------------------------------------------
2022-11-23T01:35:03.2351987Z   test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:03.2352364Z   test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:03.2352846Z   test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:35:03.2353295Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:03.2353722Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:03.2353953Z 
2022-11-23T01:35:03.2354154Z ----------------------------------------------------------------------
2022-11-23T01:35:03.2354396Z Ran 5 tests in 0.004s
2022-11-23T01:35:03.2354496Z 
2022-11-23T01:35:03.2354572Z OK (skipped=5)
2022-11-23T01:35:03.2354678Z 
2022-11-23T01:35:03.2354765Z Generating XML reports...
2022-11-23T01:35:03.2355237Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123013503.xml
2022-11-23T01:35:03.2355843Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123013503.xml
2022-11-23T01:35:03.2356140Z 
2022-11-23T01:35:03.2356414Z ##[endgroup]
2022-11-23T01:35:03.2356880Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_hdew70jd)
2022-11-23T01:35:03.2357143Z 
2022-11-23T01:35:03.2357374Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-11-23 01:35:03.235188]
2022-11-23T01:35:03.2357911Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:03.235522]
2022-11-23T01:35:04.6685101Z 
2022-11-23T01:35:04.6685703Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:35:04.6686591Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_5aqsxs5i)
2022-11-23T01:35:04.6687799Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-11-23T01:35:04.6688167Z 
2022-11-23T01:35:04.6688264Z Running tests...
2022-11-23T01:35:04.6688689Z ----------------------------------------------------------------------
2022-11-23T01:35:04.6689267Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:04.6690025Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-11-23T01:35:04.6690719Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:04.6690950Z 
2022-11-23T01:35:04.6691198Z ----------------------------------------------------------------------
2022-11-23T01:35:04.6691656Z Ran 3 tests in 0.005s
2022-11-23T01:35:04.6691872Z 
2022-11-23T01:35:04.6692024Z OK (skipped=3)
2022-11-23T01:35:04.6692230Z 
2022-11-23T01:35:04.6692345Z Generating XML reports...
2022-11-23T01:35:04.6693268Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123013504.xml
2022-11-23T01:35:04.6693763Z 
2022-11-23T01:35:04.6694010Z ##[endgroup]
2022-11-23T01:35:04.6694519Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_5aqsxs5i)
2022-11-23T01:35:04.6694816Z 
2022-11-23T01:35:04.6695199Z Running distributed/test_data_parallel ... [2022-11-23 01:35:04.668743]
2022-11-23T01:35:04.6695705Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:04.669102]
2022-11-23T01:35:06.4557359Z 
2022-11-23T01:35:06.4557994Z Expand the folded group to see the log file of distributed/test_data_parallel
2022-11-23T01:35:06.4559135Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_66ewwg9_)
2022-11-23T01:35:06.4559829Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-11-23T01:35:06.4560030Z 
2022-11-23T01:35:06.4560106Z Running tests...
2022-11-23T01:35:06.4560456Z ----------------------------------------------------------------------
2022-11-23T01:35:06.4560870Z   test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4561463Z   test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4562099Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4562466Z   test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s)
2022-11-23T01:35:06.4562936Z   test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4563716Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4564583Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4565170Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-11-23T01:35:06.4565860Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s)
2022-11-23T01:35:06.4566466Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4566927Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4567438Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2022-11-23T01:35:06.4568056Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4568509Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4568937Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4569363Z   test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4569799Z   test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4570298Z   test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4570700Z   test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4571133Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4578730Z   test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4579547Z   test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4580196Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4580953Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.000s)
2022-11-23T01:35:06.4581956Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4582868Z   test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4583782Z   test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4584716Z   test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4585596Z   test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4586491Z   test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-11-23T01:35:06.4587403Z   test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s)
2022-11-23T01:35:06.4588282Z   test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-11-23T01:35:06.4589073Z   test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.002s)
2022-11-23T01:35:06.4589977Z   test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4590858Z   test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4591770Z   test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4592685Z   test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4593613Z   test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4594576Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4595555Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4596513Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4597501Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4598658Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4599630Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4600619Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4601612Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4602682Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-11-23T01:35:06.4603209Z 
2022-11-23T01:35:06.4603644Z ----------------------------------------------------------------------
2022-11-23T01:35:06.4604145Z Ran 46 tests in 0.046s
2022-11-23T01:35:06.4604386Z 
2022-11-23T01:35:06.4604540Z OK (skipped=46)
2022-11-23T01:35:06.4604771Z 
2022-11-23T01:35:06.4604953Z Generating XML reports...
2022-11-23T01:35:06.4605923Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123013506.xml
2022-11-23T01:35:06.4607231Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20221123013506.xml
2022-11-23T01:35:06.4608026Z 
2022-11-23T01:35:06.4608707Z ##[endgroup]
2022-11-23T01:35:06.4609692Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_66ewwg9_)
2022-11-23T01:35:06.4610264Z 
2022-11-23T01:35:06.4610670Z Running distributed/test_launcher ... [2022-11-23 01:35:06.456080]
2022-11-23T01:35:06.4611738Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:06.456463]
2022-11-23T01:35:08.5791453Z 
2022-11-23T01:35:08.5791941Z Expand the folded group to see the log file of distributed/test_launcher
2022-11-23T01:35:08.5793058Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_20etxodc)
2022-11-23T01:35:08.5794047Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2022-11-23T01:35:08.5794295Z 
2022-11-23T01:35:08.5794369Z Running tests...
2022-11-23T01:35:08.5794679Z ----------------------------------------------------------------------
2022-11-23T01:35:08.5795540Z   test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T01:35:08.5795995Z 
2022-11-23T01:35:08.5796232Z ----------------------------------------------------------------------
2022-11-23T01:35:08.5796536Z Ran 1 test in 0.252s
2022-11-23T01:35:08.5796664Z 
2022-11-23T01:35:08.5796736Z OK (skipped=1)
2022-11-23T01:35:08.5796842Z 
2022-11-23T01:35:08.5796913Z Generating XML reports...
2022-11-23T01:35:08.5797523Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123013508.xml
2022-11-23T01:35:08.5797786Z 
2022-11-23T01:35:08.5798016Z ##[endgroup]
2022-11-23T01:35:08.5798420Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_20etxodc)
2022-11-23T01:35:08.5798663Z 
2022-11-23T01:35:08.5798874Z Running distributed/_composable/test_contract ... [2022-11-23 01:35:08.579573]
2022-11-23T01:35:08.5801070Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:08.579912]
2022-11-23T01:35:10.3720901Z 
2022-11-23T01:35:10.3721441Z Expand the folded group to see the log file of distributed/_composable/test_contract
2022-11-23T01:35:10.3722695Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_lp5xck75)
2022-11-23T01:35:10.3723372Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2022-11-23T01:35:10.3723578Z 
2022-11-23T01:35:10.3723654Z Running tests...
2022-11-23T01:35:10.3724139Z ----------------------------------------------------------------------
2022-11-23T01:35:10.3724409Z   test_add_hooks (__main__.TestContract) ... ok (0.264s)
2022-11-23T01:35:10.3724667Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2022-11-23T01:35:10.3724960Z   test_state (__main__.TestContract) ... ok (0.002s)
2022-11-23T01:35:10.3725126Z 
2022-11-23T01:35:10.3725354Z ----------------------------------------------------------------------
2022-11-23T01:35:10.3725608Z Ran 3 tests in 0.268s
2022-11-23T01:35:10.3725720Z 
2022-11-23T01:35:10.3725781Z OK
2022-11-23T01:35:10.3725871Z 
2022-11-23T01:35:10.3725943Z Generating XML reports...
2022-11-23T01:35:10.3726397Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123013509.xml
2022-11-23T01:35:10.3726648Z 
2022-11-23T01:35:10.3726876Z ##[endgroup]
2022-11-23T01:35:10.3727390Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_lp5xck75)
2022-11-23T01:35:10.3727666Z 
2022-11-23T01:35:10.3727882Z Running distributed/elastic/utils/logging_test ... [2022-11-23 01:35:10.372383]
2022-11-23T01:35:10.3729105Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:10.372723]
2022-11-23T01:35:12.1898907Z 
2022-11-23T01:35:12.1899484Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2022-11-23T01:35:12.1900709Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_54082vib)
2022-11-23T01:35:12.1901448Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-11-23T01:35:12.1901671Z 
2022-11-23T01:35:12.1901745Z Running tests...
2022-11-23T01:35:12.1902060Z ----------------------------------------------------------------------
2022-11-23T01:35:12.1902367Z   test_derive_module_name (__main__.LoggingTest) ... ok (0.276s)
2022-11-23T01:35:12.1902753Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-11-23T01:35:12.1902934Z 
2022-11-23T01:35:12.1903242Z ----------------------------------------------------------------------
2022-11-23T01:35:12.1903480Z Ran 2 tests in 0.279s
2022-11-23T01:35:12.1903596Z 
2022-11-23T01:35:12.1903661Z OK
2022-11-23T01:35:12.1903752Z 
2022-11-23T01:35:12.1903836Z Generating XML reports...
2022-11-23T01:35:12.1904302Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123013511.xml
2022-11-23T01:35:12.1904550Z 
2022-11-23T01:35:12.1904784Z ##[endgroup]
2022-11-23T01:35:12.1905264Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_54082vib)
2022-11-23T01:35:12.1905578Z 
2022-11-23T01:35:12.1905773Z Running distributed/fsdp/test_utils ... [2022-11-23 01:35:12.190173]
2022-11-23T01:35:12.1906622Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:12.190513]
2022-11-23T01:35:13.9793271Z 
2022-11-23T01:35:13.9793790Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2022-11-23T01:35:13.9795526Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_ndadxn5r)
2022-11-23T01:35:13.9796293Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-11-23T01:35:13.9796585Z 
2022-11-23T01:35:13.9796698Z Running tests...
2022-11-23T01:35:13.9797123Z ----------------------------------------------------------------------
2022-11-23T01:35:13.9797524Z   test_module_wrap_policy (__main__.TestGetSubmoduleToStates)
2022-11-23T01:35:13.9797888Z Tests the module wrap policy on a nested model with buffers and a ... ok (0.261s)
2022-11-23T01:35:13.9798333Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2022-11-23T01:35:13.9798722Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-11-23T01:35:13.9799129Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2022-11-23T01:35:13.9799456Z   test_packed_sequence (__main__.TestUtils)
2022-11-23T01:35:13.9799749Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-11-23T01:35:13.9800072Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-11-23T01:35:13.9800262Z 
2022-11-23T01:35:13.9800461Z ----------------------------------------------------------------------
2022-11-23T01:35:13.9800734Z Ran 6 tests in 0.272s
2022-11-23T01:35:13.9800836Z 
2022-11-23T01:35:13.9800950Z OK (skipped=2)
2022-11-23T01:35:13.9801066Z 
2022-11-23T01:35:13.9801150Z Generating XML reports...
2022-11-23T01:35:13.9801680Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123013513.xml
2022-11-23T01:35:13.9802284Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123013513.xml
2022-11-23T01:35:13.9802521Z 
2022-11-23T01:35:13.9802779Z ##[endgroup]
2022-11-23T01:35:13.9803270Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_ndadxn5r)
2022-11-23T01:35:13.9803520Z 
2022-11-23T01:35:13.9803827Z Running distributed/test_multi_threaded_pg ... [2022-11-23 01:35:13.979571]
2022-11-23T01:35:13.9804353Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:13.979910]
2022-11-23T01:35:15.7871985Z 
2022-11-23T01:35:15.7872666Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg
2022-11-23T01:35:15.7873821Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_jtrov28j)
2022-11-23T01:35:15.7874886Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg
2022-11-23T01:35:15.7875233Z 
2022-11-23T01:35:15.7875311Z Running tests...
2022-11-23T01:35:15.7875630Z ----------------------------------------------------------------------
2022-11-23T01:35:15.7876057Z   test_allgather (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7876478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7876918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7877407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7878004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7878510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7879220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7879743Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7880030Z ok (0.270s)
2022-11-23T01:35:15.7880452Z   test_broadcast (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7880929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7881347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7881915Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7882298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7882788Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7883302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7883803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7884084Z ok (0.016s)
2022-11-23T01:35:15.7884444Z   test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7884892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7885240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7885735Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7886131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7886619Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7887121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7887708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7888014Z 2 -> 4
2022-11-23T01:35:15.7888170Z 3 -> 4
2022-11-23T01:35:15.7888338Z 0 -> 4
2022-11-23T01:35:15.7888504Z 1 -> 4
2022-11-23T01:35:15.7888642Z ok (0.014s)
2022-11-23T01:35:15.7888995Z   test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7889437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7889801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7890145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7890632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7891147Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7891663Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7892158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7892489Z ok (0.013s)
2022-11-23T01:35:15.7892836Z   test_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7893268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7893617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7893975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7894505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7895184Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7895711Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7896222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7896508Z ok (0.012s)
2022-11-23T01:35:15.7896851Z   test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:15.7897294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:15.7897659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:15.7898149Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7898526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:15.7899013Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7899523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7900038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:15.7900305Z ok (0.014s)
2022-11-23T01:35:15.7900410Z 
2022-11-23T01:35:15.7900610Z ----------------------------------------------------------------------
2022-11-23T01:35:15.7900855Z Ran 6 tests in 0.341s
2022-11-23T01:35:15.7900972Z 
2022-11-23T01:35:15.7901019Z OK
2022-11-23T01:35:15.7901110Z 
2022-11-23T01:35:15.7901195Z Generating XML reports...
2022-11-23T01:35:15.7901686Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123013515.xml
2022-11-23T01:35:15.7902326Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123013515.xml
2022-11-23T01:35:15.7902593Z 
2022-11-23T01:35:15.7902870Z ##[endgroup]
2022-11-23T01:35:15.7903325Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_jtrov28j)
2022-11-23T01:35:15.7903582Z 
2022-11-23T01:35:15.7903799Z Running distributed/fsdp/test_fsdp_traversal ... [2022-11-23 01:35:15.787628]
2022-11-23T01:35:15.7904308Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:15.787983]
2022-11-23T01:35:19.2250413Z 
2022-11-23T01:35:19.2250828Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2022-11-23T01:35:19.2251632Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_4kfxplap)
2022-11-23T01:35:19.2252526Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-11-23T01:35:19.2252789Z 
2022-11-23T01:35:19.2252864Z Running tests...
2022-11-23T01:35:19.2253162Z ----------------------------------------------------------------------
2022-11-23T01:35:19.2253591Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25604
2022-11-23T01:35:19.2254117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25605
2022-11-23T01:35:19.2254751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:19.2255366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:19.2255804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:19.2256156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:19.2256483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:19.2256953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:19.2257283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:19.2257758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:19.2258281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:19.2258660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:19.2259162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:19.2259682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:19.2260056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:19.2260402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:19.2260664Z dist init r=1, world=2
2022-11-23T01:35:19.2260829Z dist init r=0, world=2
2022-11-23T01:35:19.2261033Z skip: Need at least 2 CUDA devices (1.872s)
2022-11-23T01:35:19.2261169Z 
2022-11-23T01:35:19.2261367Z ----------------------------------------------------------------------
2022-11-23T01:35:19.2261593Z Ran 1 test in 1.872s
2022-11-23T01:35:19.2261704Z 
2022-11-23T01:35:19.2261774Z OK (skipped=1)
2022-11-23T01:35:19.2261881Z 
2022-11-23T01:35:19.2261965Z Generating XML reports...
2022-11-23T01:35:19.2262406Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123013516.xml
2022-11-23T01:35:19.2262659Z 
2022-11-23T01:35:19.2262904Z ##[endgroup]
2022-11-23T01:35:19.2263369Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_4kfxplap)
2022-11-23T01:35:19.2263633Z 
2022-11-23T01:35:19.2263870Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-11-23 01:35:19.225292]
2022-11-23T01:35:19.2264411Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:19.225692]
2022-11-23T01:35:22.7891790Z 
2022-11-23T01:35:22.7892411Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T01:35:22.7893782Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_thoh0bg1)
2022-11-23T01:35:22.7895083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi69haimn
2022-11-23T01:35:22.7895509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi69haimn/_remote_module_non_scriptable.py
2022-11-23T01:35:22.7896116Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-11-23T01:35:22.7896394Z 
2022-11-23T01:35:22.7896455Z Running tests...
2022-11-23T01:35:22.7896765Z ----------------------------------------------------------------------
2022-11-23T01:35:22.7897264Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25643
2022-11-23T01:35:22.7898150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25644
2022-11-23T01:35:22.7898501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25645
2022-11-23T01:35:22.7898829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25646
2022-11-23T01:35:22.7899306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:22.7899629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:22.7900063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:22.7900411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:22.7900893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77l544sk
2022-11-23T01:35:22.7901566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77l544sk/_remote_module_non_scriptable.py
2022-11-23T01:35:22.7901946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:35:22.7902409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:22.7902730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:22.7903161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:22.7903506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:22.7903847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqn6p1e4
2022-11-23T01:35:22.7904234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqn6p1e4/_remote_module_non_scriptable.py
2022-11-23T01:35:22.7904610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:35:22.7905070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:22.7905395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:22.7905811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:22.7906156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:22.7906496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1kjvmcb
2022-11-23T01:35:22.7906883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1kjvmcb/_remote_module_non_scriptable.py
2022-11-23T01:35:22.7907367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:22.7907694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:22.7908116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:22.7908447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:22.7908789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9w_4j92
2022-11-23T01:35:22.7909260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9w_4j92/_remote_module_non_scriptable.py
2022-11-23T01:35:22.7909624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:22.7909968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:22.7910213Z ok (1.978s)
2022-11-23T01:35:22.7910462Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-11-23T01:35:22.7910792Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:22.7911158Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-11-23T01:35:22.7911499Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:35:22.7911811Z   test_check_overlapping (__main__.TestShardingSpec) ... ok (0.003s)
2022-11-23T01:35:22.7912137Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2022-11-23T01:35:22.7912482Z   test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2022-11-23T01:35:22.7912830Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.004s)
2022-11-23T01:35:22.7913145Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-11-23T01:35:22.7913442Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-11-23T01:35:22.7913723Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-11-23T01:35:22.7914019Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.007s)
2022-11-23T01:35:22.7914203Z 
2022-11-23T01:35:22.7914410Z ----------------------------------------------------------------------
2022-11-23T01:35:22.7914652Z Ran 11 tests in 2.000s
2022-11-23T01:35:22.7914767Z 
2022-11-23T01:35:22.7914837Z OK (skipped=5)
2022-11-23T01:35:22.7914932Z 
2022-11-23T01:35:22.7915017Z Generating XML reports...
2022-11-23T01:35:22.7915510Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123013520.xml
2022-11-23T01:35:22.7916146Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123013520.xml
2022-11-23T01:35:22.7916409Z 
2022-11-23T01:35:22.7916650Z ##[endgroup]
2022-11-23T01:35:22.7917156Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_thoh0bg1)
2022-11-23T01:35:22.7917452Z 
2022-11-23T01:35:22.7917685Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-11-23 01:35:22.789547]
2022-11-23T01:35:22.7918242Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:22.789909]
2022-11-23T01:35:26.4381861Z 
2022-11-23T01:35:26.4382496Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:35:26.4383763Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_4hf1_79z)
2022-11-23T01:35:26.4384885Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-11-23T01:35:26.4385235Z 
2022-11-23T01:35:26.4385337Z Running tests...
2022-11-23T01:35:26.4385831Z ----------------------------------------------------------------------
2022-11-23T01:35:26.4386302Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-11-23T01:35:26.4386961Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25698
2022-11-23T01:35:26.4387846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25699
2022-11-23T01:35:26.4388423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25700
2022-11-23T01:35:26.4388988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25701
2022-11-23T01:35:26.4389730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:26.4390247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:26.4390983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:26.4391680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:26.4392170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:26.4392829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:26.4393285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:26.4393911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:26.4394406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:26.4394970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:26.4395711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:26.4396215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:26.4396875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:26.4397410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:26.4397866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:26.4398532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:26.4398987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:26.4399555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:26.4400038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:26.4400500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:26.4401197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:26.4401925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:26.4402669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:26.4403396Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:26.4403971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:35:26.4404461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:26.4404952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:26.4405464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:35:26.4405822Z dist init r=2, world=4
2022-11-23T01:35:26.4406087Z dist init r=0, world=4
2022-11-23T01:35:26.4406354Z dist init r=1, world=4
2022-11-23T01:35:26.4406600Z dist init r=3, world=4
2022-11-23T01:35:26.4406898Z skip: Need at least 2 CUDA devices (2.080s)
2022-11-23T01:35:26.4407189Z 
2022-11-23T01:35:26.4407578Z ----------------------------------------------------------------------
2022-11-23T01:35:26.4407925Z Ran 1 test in 2.080s
2022-11-23T01:35:26.4408076Z 
2022-11-23T01:35:26.4408181Z OK (skipped=1)
2022-11-23T01:35:26.4408339Z 
2022-11-23T01:35:26.4408462Z Generating XML reports...
2022-11-23T01:35:26.4409181Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123013524.xml
2022-11-23T01:35:26.4409578Z 
2022-11-23T01:35:26.4409982Z ##[endgroup]
2022-11-23T01:35:26.4410730Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_4hf1_79z)
2022-11-23T01:35:26.4411213Z 
2022-11-23T01:35:26.4411541Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-11-23 01:35:26.438515]
2022-11-23T01:35:26.4412359Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:26.438958]
2022-11-23T01:35:30.0794446Z 
2022-11-23T01:35:30.0795239Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2022-11-23T01:35:30.0796395Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward__9_vbhna)
2022-11-23T01:35:30.0797530Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-11-23T01:35:30.0797931Z 
2022-11-23T01:35:30.0798070Z Running tests...
2022-11-23T01:35:30.0798588Z ----------------------------------------------------------------------
2022-11-23T01:35:30.0799191Z   test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25765
2022-11-23T01:35:30.0799869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25766
2022-11-23T01:35:30.0800479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25767
2022-11-23T01:35:30.0801073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25768
2022-11-23T01:35:30.0801945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:30.0802532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:30.0803031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:30.0803698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:30.0804068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:30.0804528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:30.0804855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:30.0805277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:30.0805603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:30.0805938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:30.0806406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:30.0806730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:30.0807139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:30.0807562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:30.0808088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:35:30.0808546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:30.0808868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:30.0809287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:30.0809626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:30.0809943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:35:30.0810498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:30.0811023Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:30.0811541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:30.0812042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:35:30.0812427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:35:30.0812772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:30.0813114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:35:30.0813446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:30.0813705Z dist init r=2, world=4
2022-11-23T01:35:30.0813885Z dist init r=0, world=4
2022-11-23T01:35:30.0814046Z dist init r=3, world=4
2022-11-23T01:35:30.0814219Z dist init r=1, world=4
2022-11-23T01:35:30.0814423Z skip: Need at least 2 CUDA devices (2.076s)
2022-11-23T01:35:30.0814558Z 
2022-11-23T01:35:30.0814746Z ----------------------------------------------------------------------
2022-11-23T01:35:30.0815177Z Ran 1 test in 2.076s
2022-11-23T01:35:30.0815290Z 
2022-11-23T01:35:30.0815363Z OK (skipped=1)
2022-11-23T01:35:30.0815468Z 
2022-11-23T01:35:30.0815539Z Generating XML reports...
2022-11-23T01:35:30.0816008Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123013527.xml
2022-11-23T01:35:30.0816274Z 
2022-11-23T01:35:30.0816548Z ##[endgroup]
2022-11-23T01:35:30.0817017Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward__9_vbhna)
2022-11-23T01:35:30.0817296Z 
2022-11-23T01:35:30.0817498Z Running distributed/rpc/test_share_memory ... [2022-11-23 01:35:30.079758]
2022-11-23T01:35:30.0818035Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:30.080134]
2022-11-23T01:35:35.1308798Z 
2022-11-23T01:35:35.1309392Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2022-11-23T01:35:35.1314050Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_r9mg9hjq)
2022-11-23T01:35:35.1314447Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:35:35.1314737Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-11-23T01:35:35.1315037Z test_case (__main__.TestRPCPickler)
2022-11-23T01:35:35.1315511Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-11-23T01:35:35.1316008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:35.1316436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:35.1317135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:35.1317559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:35.1317727Z 
2022-11-23T01:35:35.1317800Z Running tests...
2022-11-23T01:35:35.1318094Z ----------------------------------------------------------------------
2022-11-23T01:35:35.1318606Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:35.1319034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:35.1319454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:35.1319806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:35.1320116Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f9e0cf14710>
2022-11-23T01:35:35.1320375Z Traceback (most recent call last):
2022-11-23T01:35:35.1320747Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-11-23T01:35:35.1321177Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 767, in _free_weak_ref
2022-11-23T01:35:35.1321549Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-11-23T01:35:35.1321835Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f9e0cf14710>
2022-11-23T01:35:35.1322091Z Traceback (most recent call last):
2022-11-23T01:35:35.1322471Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-11-23T01:35:35.1322894Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 767, in _free_weak_ref
2022-11-23T01:35:35.1323246Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-11-23T01:35:35.1323891Z /opt/conda/lib/python3.7/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T01:35:35.1324332Z   warnings.warn(message, UserWarning)
2022-11-23T01:35:35.1324527Z ok (2.069s)
2022-11-23T01:35:35.1324616Z 
2022-11-23T01:35:35.1324813Z ----------------------------------------------------------------------
2022-11-23T01:35:35.1325051Z Ran 1 test in 2.069s
2022-11-23T01:35:35.1325161Z 
2022-11-23T01:35:35.1325220Z OK
2022-11-23T01:35:35.1325311Z 
2022-11-23T01:35:35.1325384Z Generating XML reports...
2022-11-23T01:35:35.1325823Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123013532.xml
2022-11-23T01:35:35.1326073Z 
2022-11-23T01:35:35.1326313Z ##[endgroup]
2022-11-23T01:35:35.1326748Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_r9mg9hjq)
2022-11-23T01:35:35.1327005Z 
2022-11-23T01:35:35.1327219Z Running distributed/elastic/utils/distributed_test ... [2022-11-23 01:35:35.131188]
2022-11-23T01:35:35.1327839Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:35.131547]
2022-11-23T01:35:40.0379447Z 
2022-11-23T01:35:40.0380014Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test
2022-11-23T01:35:40.0381304Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_dbxj4yl_)
2022-11-23T01:35:40.0382289Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-11-23T01:35:40.0382507Z 
2022-11-23T01:35:40.0382585Z Running tests...
2022-11-23T01:35:40.0383166Z ----------------------------------------------------------------------
2022-11-23T01:35:40.0383745Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.267s)
2022-11-23T01:35:40.0384082Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-11-23T01:35:40.0384953Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.000s)
2022-11-23T01:35:40.0385618Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.014s)
2022-11-23T01:35:40.0386030Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (5347e8488c91, 0).
2022-11-23T01:35:40.0386394Z ok (0.001s)
2022-11-23T01:35:40.0386871Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:44027 (errno: 98 - Address already in use).
2022-11-23T01:35:40.0387388Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:44027 (errno: 98 - Address already in use).
2022-11-23T01:35:40.0387731Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-11-23T01:35:40.0387971Z ok (0.003s)
2022-11-23T01:35:40.0388292Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (5347e8488c91, 43263).
2022-11-23T01:35:40.0388605Z ok (0.002s)
2022-11-23T01:35:40.0388704Z 
2022-11-23T01:35:40.0388903Z ----------------------------------------------------------------------
2022-11-23T01:35:40.0389131Z Ran 7 tests in 3.290s
2022-11-23T01:35:40.0389243Z 
2022-11-23T01:35:40.0389315Z OK (skipped=1)
2022-11-23T01:35:40.0389423Z 
2022-11-23T01:35:40.0389507Z Generating XML reports...
2022-11-23T01:35:40.0389994Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123013536.xml
2022-11-23T01:35:40.0390293Z 
2022-11-23T01:35:40.0390517Z ##[endgroup]
2022-11-23T01:35:40.0391014Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_dbxj4yl_)
2022-11-23T01:35:40.0391312Z 
2022-11-23T01:35:40.0391505Z Running distributed/fsdp/test_fsdp_input ... [2022-11-23 01:35:40.038233]
2022-11-23T01:35:40.0392009Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:40.038589]
2022-11-23T01:35:44.9360810Z 
2022-11-23T01:35:44.9361564Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input
2022-11-23T01:35:44.9362421Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_ip32uipk)
2022-11-23T01:35:44.9363017Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-11-23T01:35:44.9363222Z 
2022-11-23T01:35:44.9363320Z Running tests...
2022-11-23T01:35:44.9363634Z ----------------------------------------------------------------------
2022-11-23T01:35:44.9363912Z   test_input_type_dict (__main__.TestInput)
2022-11-23T01:35:44.9364252Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25969
2022-11-23T01:35:44.9364777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:44.9365204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:44.9365949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:44.9366442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:44.9366785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:44.9367375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:35:44.9367768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:44.9368018Z dist init r=0, world=1
2022-11-23T01:35:44.9368310Z skip: Need at least 1 CUDA device (1.777s)
2022-11-23T01:35:44.9368546Z   test_input_type_list (__main__.TestInput)
2022-11-23T01:35:44.9368883Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25983
2022-11-23T01:35:44.9369404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:44.9369888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:44.9370412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:44.9370759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:44.9371096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:44.9371595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:35:44.9371972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:44.9372231Z dist init r=0, world=1
2022-11-23T01:35:44.9372435Z skip: Need at least 1 CUDA device (1.509s)
2022-11-23T01:35:44.9372573Z 
2022-11-23T01:35:44.9372774Z ----------------------------------------------------------------------
2022-11-23T01:35:44.9373003Z Ran 2 tests in 3.286s
2022-11-23T01:35:44.9373115Z 
2022-11-23T01:35:44.9373186Z OK (skipped=2)
2022-11-23T01:35:44.9373294Z 
2022-11-23T01:35:44.9373377Z Generating XML reports...
2022-11-23T01:35:44.9373790Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123013541.xml
2022-11-23T01:35:44.9374031Z 
2022-11-23T01:35:44.9374304Z ##[endgroup]
2022-11-23T01:35:44.9374755Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_ip32uipk)
2022-11-23T01:35:44.9375174Z 
2022-11-23T01:35:44.9375411Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-11-23 01:35:44.936446]
2022-11-23T01:35:44.9375950Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:44.936898]
2022-11-23T01:35:50.0380098Z 
2022-11-23T01:35:50.0380938Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2022-11-23T01:35:50.0382214Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_xprjj8ob)
2022-11-23T01:35:50.0383308Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-11-23T01:35:50.0383633Z 
2022-11-23T01:35:50.0383784Z Running tests...
2022-11-23T01:35:50.0384286Z ----------------------------------------------------------------------
2022-11-23T01:35:50.0385050Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26008
2022-11-23T01:35:50.0385792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26009
2022-11-23T01:35:50.0386846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:50.0387403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:50.0388090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:50.0388647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:50.0389199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:50.0389936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:50.0390558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:50.0391160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:50.0391651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:50.0392114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:50.0392850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:50.0393683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:50.0394305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:50.0394822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:50.0395234Z dist init r=1, world=2
2022-11-23T01:35:50.0395501Z dist init r=0, world=2
2022-11-23T01:35:50.0395793Z skip: Need at least 2 CUDA devices (1.878s)
2022-11-23T01:35:50.0396391Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26036
2022-11-23T01:35:50.0397028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26037
2022-11-23T01:35:50.0397694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:50.0398146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:50.0398760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:50.0399247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:50.0399706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:35:50.0400383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:35:50.0400849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:35:50.0401446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:35:50.0401943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:35:50.0402426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:35:50.0403165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:50.0403926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:35:50.0404516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:35:50.0405026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:35:50.0405397Z dist init r=1, world=2
2022-11-23T01:35:50.0405732Z dist init r=0, world=2
2022-11-23T01:35:50.0406049Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:35:50.0406247Z 
2022-11-23T01:35:50.0406540Z ----------------------------------------------------------------------
2022-11-23T01:35:50.0406867Z Ran 2 tests in 3.492s
2022-11-23T01:35:50.0407027Z 
2022-11-23T01:35:50.0407129Z OK (skipped=2)
2022-11-23T01:35:50.0407368Z 
2022-11-23T01:35:50.0407487Z Generating XML reports...
2022-11-23T01:35:50.0408237Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123013546.xml
2022-11-23T01:35:50.0408724Z 
2022-11-23T01:35:50.0409121Z ##[endgroup]
2022-11-23T01:35:50.0409882Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_xprjj8ob)
2022-11-23T01:35:50.0410293Z 
2022-11-23T01:35:50.0410628Z Running distributed/elastic/timer/local_timer_test ... [2022-11-23 01:35:50.038347]
2022-11-23T01:35:50.0411449Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:50.038777]
2022-11-23T01:35:55.4706816Z 
2022-11-23T01:35:55.4707343Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test
2022-11-23T01:35:55.4708407Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_34rfvhnr)
2022-11-23T01:35:55.4709379Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test
2022-11-23T01:35:55.4709700Z 
2022-11-23T01:35:55.4709791Z Running tests...
2022-11-23T01:35:55.4710244Z ----------------------------------------------------------------------
2022-11-23T01:35:55.4710669Z   test_acquire_release (__main__.LocalTimerServerTest)
2022-11-23T01:35:55.4711866Z tests that: ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/87154 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s)
2022-11-23T01:35:55.4712775Z   test_expired_timers (__main__.LocalTimerServerTest)
2022-11-23T01:35:55.4713243Z tests that a single expired timer on a process should terminate ... ok (0.003s)
2022-11-23T01:35:55.4713722Z   test_valid_timers (__main__.LocalTimerServerTest)
2022-11-23T01:35:55.4714231Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s)
2022-11-23T01:35:55.4714739Z   test_watchdog_call_count (__main__.LocalTimerServerTest)
2022-11-23T01:35:55.4715382Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.103s)
2022-11-23T01:35:55.4715860Z   test_watchdog_empty_queue (__main__.LocalTimerServerTest)
2022-11-23T01:35:55.4716343Z checks that the watchdog can run on an empty queue ... ok (0.011s)
2022-11-23T01:35:55.4716817Z   test_client_interaction (__main__.LocalTimerTest) ... ok (0.007s)
2022-11-23T01:35:55.4717308Z   test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s)
2022-11-23T01:35:55.4717806Z   test_get_timer_recursive (__main__.LocalTimerTest)
2022-11-23T01:35:55.4718304Z If a function acquires a countdown timer with default scope, ... ok (1.688s)
2022-11-23T01:35:55.4718863Z   test_happy_path (__main__.LocalTimerTest) ... ok (0.103s)
2022-11-23T01:35:55.4719342Z   test_no_client (__main__.LocalTimerTest) ... ok (0.011s)
2022-11-23T01:35:55.4719731Z   test_timer (__main__.LocalTimerTest) ... ok (0.129s)
2022-11-23T01:35:55.4720194Z   test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.022s)
2022-11-23T01:35:55.4720712Z   test_get_less_than_size (__main__.MultiprocessingRequestQueueTest)
2022-11-23T01:35:55.4721204Z Tests slow producer. ... ok (0.507s)
2022-11-23T01:35:55.4721933Z   test_get_size (__main__.MultiprocessingRequestQueueTest)
2022-11-23T01:35:55.4722495Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.910s)
2022-11-23T01:35:55.4722792Z 
2022-11-23T01:35:55.4723188Z ----------------------------------------------------------------------
2022-11-23T01:35:55.4723591Z Ran 14 tests in 3.772s
2022-11-23T01:35:55.4723780Z 
2022-11-23T01:35:55.4723902Z OK (skipped=1)
2022-11-23T01:35:55.4724084Z 
2022-11-23T01:35:55.4724232Z Generating XML reports...
2022-11-23T01:35:55.4725116Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123013551.xml
2022-11-23T01:35:55.4726364Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123013551.xml
2022-11-23T01:35:55.4727650Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123013551.xml
2022-11-23T01:35:55.4728233Z 
2022-11-23T01:35:55.4728719Z ##[endgroup]
2022-11-23T01:35:55.4729646Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_34rfvhnr)
2022-11-23T01:35:55.4730147Z 
2022-11-23T01:35:55.4730590Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-11-23 01:35:55.471009]
2022-11-23T01:35:55.4731566Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:35:55.471382]
2022-11-23T01:36:00.7639706Z 
2022-11-23T01:36:00.7640479Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:36:00.7641771Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_j65o1zt4)
2022-11-23T01:36:00.7642957Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-11-23T01:36:00.7643182Z 
2022-11-23T01:36:00.7643256Z Running tests...
2022-11-23T01:36:00.7643565Z ----------------------------------------------------------------------
2022-11-23T01:36:00.7643881Z   test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s)
2022-11-23T01:36:00.7644243Z   test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7644619Z   test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:36:00.7645043Z   test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7645497Z   test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7646045Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:36:00.7646459Z   test_shard_metadata_init (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7646838Z   test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7647216Z   test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7647680Z   test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7648054Z   test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7648433Z   test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7648991Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7649353Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7649723Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7650160Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7650665Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7651064Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7651387Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7651693Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7652032Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7652363Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7652678Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7652978Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7653321Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7653654Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-11-23T01:36:00.7662336Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7662783Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7663203Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7663618Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7664030Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7664444Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7664832Z   test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7665232Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7665639Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7666171Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7666565Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7666961Z   test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7667357Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7667773Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7668172Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7668571Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:36:00.7669088Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7669464Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-11-23T01:36:00.7669786Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7670093Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-11-23T01:36:00.7670444Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7670832Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-11-23T01:36:00.7671186Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7671598Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7672024Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7672421Z   test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7672829Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7673253Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7673685Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7674097Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7674519Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7674944Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7675354Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7675749Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7676174Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7676647Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7677264Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-11-23T01:36:00.7677749Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7678262Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26117
2022-11-23T01:36:00.7678701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26118
2022-11-23T01:36:00.7679036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26119
2022-11-23T01:36:00.7679353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26120
2022-11-23T01:36:00.7679872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7680245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7680682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7681015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7681335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:00.7681796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7682108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7682567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7682908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7683229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:00.7683674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7684001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7684432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7684772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7685079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:00.7685535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7685860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7686273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7686613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7686929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:00.7687366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:00.7687726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:00.7688094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:00.7688455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:00.7688940Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:00.7689465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:00.7689981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:00.7690498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:00.7690921Z skip: Need at least 4 CUDA devices (1.976s)
2022-11-23T01:36:00.7691629Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:36:00.7692159Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7692653Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7693159Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7693608Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7694105Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26173
2022-11-23T01:36:00.7694542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26174
2022-11-23T01:36:00.7695012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26175
2022-11-23T01:36:00.7695405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26176
2022-11-23T01:36:00.7695859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7696195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7696613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7696959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7697283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:00.7697739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7698050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7698478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7698819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7699126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:00.7699585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7699910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7700332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7700661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7700980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:00.7701430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:00.7701743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:00.7702165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:00.7702504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:00.7702823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:00.7703100Z skip: Need at least 4 CUDA devices (1.619s)
2022-11-23T01:36:00.7703446Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7703903Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:36:00.7704300Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.035s)
2022-11-23T01:36:00.7704476Z 
2022-11-23T01:36:00.7704677Z ----------------------------------------------------------------------
2022-11-23T01:36:00.7704917Z Ran 64 tests in 3.704s
2022-11-23T01:36:00.7705029Z 
2022-11-23T01:36:00.7705101Z OK (skipped=63)
2022-11-23T01:36:00.7705209Z 
2022-11-23T01:36:00.7705280Z Generating XML reports...
2022-11-23T01:36:00.7705828Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123013556.xml
2022-11-23T01:36:00.7706503Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123013556.xml
2022-11-23T01:36:00.7707135Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123013556.xml
2022-11-23T01:36:00.7707730Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123013556.xml
2022-11-23T01:36:00.7708373Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123013556.xml
2022-11-23T01:36:00.7708990Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123013556.xml
2022-11-23T01:36:00.7709605Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123013556.xml
2022-11-23T01:36:00.7710225Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123013556.xml
2022-11-23T01:36:00.7710893Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123013556.xml
2022-11-23T01:36:00.7711560Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123013556.xml
2022-11-23T01:36:00.7712245Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123013556.xml
2022-11-23T01:36:00.7712941Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123013556.xml
2022-11-23T01:36:00.7713249Z 
2022-11-23T01:36:00.7713545Z ##[endgroup]
2022-11-23T01:36:00.7714053Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_j65o1zt4)
2022-11-23T01:36:00.7714349Z 
2022-11-23T01:36:00.7714562Z Running distributed/fsdp/test_fsdp_apply ... [2022-11-23 01:36:00.764482]
2022-11-23T01:36:00.7715065Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:00.764945]
2022-11-23T01:36:07.4406409Z 
2022-11-23T01:36:07.4413040Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2022-11-23T01:36:07.4414248Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_b4enskbi)
2022-11-23T01:36:07.4415456Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-11-23T01:36:07.4415826Z 
2022-11-23T01:36:07.4415946Z Running tests...
2022-11-23T01:36:07.4416490Z ----------------------------------------------------------------------
2022-11-23T01:36:07.4417014Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-11-23T01:36:07.4417703Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26228
2022-11-23T01:36:07.4418403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26229
2022-11-23T01:36:07.4419230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4419774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4420817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4421469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4422081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:07.4422943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4423533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4424345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4425072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4425622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:07.4426557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4427520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4428239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:07.4428859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:07.4429331Z dist init r=0, world=2
2022-11-23T01:36:07.4429652Z dist init r=1, world=2
2022-11-23T01:36:07.4430016Z skip: Need at least 2 CUDA devices (1.869s)
2022-11-23T01:36:07.4430391Z   test_nested_module_apply (__main__.TestApply)
2022-11-23T01:36:07.4431141Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26256
2022-11-23T01:36:07.4431759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26257
2022-11-23T01:36:07.4432510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4433031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4433744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4434289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4434838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:07.4435630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4436141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4436784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4437367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4437945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:07.4438765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4439617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4440308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:07.4440958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:07.4441422Z dist init r=0, world=2
2022-11-23T01:36:07.4441737Z dist init r=1, world=2
2022-11-23T01:36:07.4442110Z skip: Need at least 2 CUDA devices (1.612s)
2022-11-23T01:36:07.4442538Z   test_transformer_module_apply (__main__.TestApply)
2022-11-23T01:36:07.4443540Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26284
2022-11-23T01:36:07.4444282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26285
2022-11-23T01:36:07.4445097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4445680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4446389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4447082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4447702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:07.4448485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:07.4449025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:07.4449754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:07.4450260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:07.4450755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:07.4451548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4452348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:07.4452971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:07.4453527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:07.4453937Z dist init r=1, world=2
2022-11-23T01:36:07.4454221Z dist init r=0, world=2
2022-11-23T01:36:07.4454540Z skip: Need at least 2 CUDA devices (1.612s)
2022-11-23T01:36:07.4454753Z 
2022-11-23T01:36:07.4455233Z ----------------------------------------------------------------------
2022-11-23T01:36:07.4455590Z Ran 3 tests in 5.094s
2022-11-23T01:36:07.4455791Z 
2022-11-23T01:36:07.4455903Z OK (skipped=3)
2022-11-23T01:36:07.4456068Z 
2022-11-23T01:36:07.4456212Z Generating XML reports...
2022-11-23T01:36:07.4456933Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123013602.xml
2022-11-23T01:36:07.4457329Z 
2022-11-23T01:36:07.4457752Z ##[endgroup]
2022-11-23T01:36:07.4458425Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_b4enskbi)
2022-11-23T01:36:07.4458692Z 
2022-11-23T01:36:07.4458917Z Running distributed/_tensor/parallel/test_tp_examples ... [2022-11-23 01:36:07.440916]
2022-11-23T01:36:07.4459466Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:07.441335]
2022-11-23T01:36:14.8422355Z 
2022-11-23T01:36:14.8422966Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_tp_examples
2022-11-23T01:36:14.8424218Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_examples_tojilpvj)
2022-11-23T01:36:14.8424912Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples
2022-11-23T01:36:14.8425151Z 
2022-11-23T01:36:14.8425271Z Running tests...
2022-11-23T01:36:14.8425589Z ----------------------------------------------------------------------
2022-11-23T01:36:14.8426264Z   test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26323
2022-11-23T01:36:14.8427009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26324
2022-11-23T01:36:14.8427629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26325
2022-11-23T01:36:14.8428245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26326
2022-11-23T01:36:14.8428955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8429323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8429904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8430255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8430591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:14.8431089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8431477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8431969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8432429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8432810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:14.8433275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8433594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8434018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8434359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8436385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:14.8437262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8437658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8438088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8438420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8438749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:14.8439113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:14.8439477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:14.8439964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8440361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:14.8440720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:14.8441345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8442179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8443115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8443407Z ok (2.168s)
2022-11-23T01:36:14.8443836Z   test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26379
2022-11-23T01:36:14.8444237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26380
2022-11-23T01:36:14.8444567Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26381
2022-11-23T01:36:14.8444895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26382
2022-11-23T01:36:14.8445352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8445667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8446139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8446486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8446796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:14.8447333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8447664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8448091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8448421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8448740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:14.8449208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8449535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8449949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8450292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8450613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:14.8451052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8451375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8451798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8452139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8452446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:14.8452807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:14.8453172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:14.8453524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:14.8453881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:14.8461476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8462588Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8463171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8463701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8464027Z skip: Need at least 4 CUDA devices (1.822s)
2022-11-23T01:36:14.8464521Z   test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26435
2022-11-23T01:36:14.8464955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26436
2022-11-23T01:36:14.8465287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26437
2022-11-23T01:36:14.8465613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26438
2022-11-23T01:36:14.8466055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8466388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8466873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8467207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8467543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:14.8468441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8468990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8469718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8470221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8470541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:14.8471004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8471321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8471750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8472097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8472407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:14.8472864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:14.8473187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:14.8473609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:14.8473934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:14.8474257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:14.8474617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:14.8474971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:14.8475334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:14.8475694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:14.8476177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8476686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8477206Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8477720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:14.8478039Z skip: Need at least 4 CUDA devices (1.823s)
2022-11-23T01:36:14.8478175Z 
2022-11-23T01:36:14.8478424Z ----------------------------------------------------------------------
2022-11-23T01:36:14.8478669Z Ran 3 tests in 5.813s
2022-11-23T01:36:14.8478781Z 
2022-11-23T01:36:14.8478852Z OK (skipped=2)
2022-11-23T01:36:14.8478959Z 
2022-11-23T01:36:14.8479032Z Generating XML reports...
2022-11-23T01:36:14.8479550Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20221123013608.xml
2022-11-23T01:36:14.8479851Z 
2022-11-23T01:36:14.8480236Z ##[endgroup]
2022-11-23T01:36:14.8480774Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_tp_examples_tojilpvj)
2022-11-23T01:36:14.8481094Z 
2022-11-23T01:36:14.8481291Z Running distributed/test_c10d_object_collectives ... [2022-11-23 01:36:14.842729]
2022-11-23T01:36:14.8481827Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:14.843116]
2022-11-23T01:36:22.8026931Z 
2022-11-23T01:36:22.8027528Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives
2022-11-23T01:36:22.8028670Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_2hy9rc6y)
2022-11-23T01:36:22.8029704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2022-11-23T01:36:22.8030091Z 
2022-11-23T01:36:22.8030221Z Running tests...
2022-11-23T01:36:22.8030790Z ----------------------------------------------------------------------
2022-11-23T01:36:22.8031483Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26502
2022-11-23T01:36:22.8032190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26503
2022-11-23T01:36:22.8032955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8034124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8034945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8035383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8035705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:22.8036178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8036493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8036970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8037461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8037798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:22.8038196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:22.8038559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:22.8039101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8039623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8039901Z ok (1.771s)
2022-11-23T01:36:22.8040226Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26530
2022-11-23T01:36:22.8040808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26531
2022-11-23T01:36:22.8041261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8041579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8042006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8042352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8042658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:22.8043169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8043490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8043912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8044242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8044560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:22.8044923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:22.8045285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:22.8045767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8046283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8046565Z ok (1.614s)
2022-11-23T01:36:22.8046868Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26558
2022-11-23T01:36:22.8047343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26559
2022-11-23T01:36:22.8047800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8048125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8048538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8048879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8049200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:22.8049645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8049971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8050399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8050741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8051045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:22.8051403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:22.8051766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:22.8052256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8052760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8053046Z ok (1.512s)
2022-11-23T01:36:22.8053401Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26586
2022-11-23T01:36:22.8053783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26587
2022-11-23T01:36:22.8054231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8054554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8055160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8055494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8055875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:22.8056338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:22.8056664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:22.8057076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:22.8057415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:22.8057736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:22.8058083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:22.8058446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:22.8058931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8059456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:22.8059728Z ok (1.512s)
2022-11-23T01:36:22.8059828Z 
2022-11-23T01:36:22.8060027Z ----------------------------------------------------------------------
2022-11-23T01:36:22.8060265Z Ran 4 tests in 6.410s
2022-11-23T01:36:22.8060376Z 
2022-11-23T01:36:22.8060422Z OK
2022-11-23T01:36:22.8060511Z 
2022-11-23T01:36:22.8060594Z Generating XML reports...
2022-11-23T01:36:22.8061061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123013616.xml
2022-11-23T01:36:22.8061331Z 
2022-11-23T01:36:22.8061602Z ##[endgroup]
2022-11-23T01:36:22.8062073Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_2hy9rc6y)
2022-11-23T01:36:22.8062345Z 
2022-11-23T01:36:22.8062552Z Running distributed/_composable/test_fully_shard ... [2022-11-23 01:36:22.803169]
2022-11-23T01:36:22.8063084Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:22.803641]
2022-11-23T01:36:32.7013778Z 
2022-11-23T01:36:32.7014302Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard
2022-11-23T01:36:32.7015638Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_iy_0ktaz)
2022-11-23T01:36:32.7016586Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard
2022-11-23T01:36:32.7016910Z 
2022-11-23T01:36:32.7017017Z Running tests...
2022-11-23T01:36:32.7017467Z ----------------------------------------------------------------------
2022-11-23T01:36:32.7017912Z   test_auto_wrap_policy (__main__.TestFSDPInitialization)
2022-11-23T01:36:32.7018429Z Tests passing an ``auto_wrap_policy``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26625
2022-11-23T01:36:32.7019048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26626
2022-11-23T01:36:32.7020088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7020618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7021334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7021894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7022432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:32.7023447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7024020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7024806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7025438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7026095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:32.7026932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7027839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7028541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:32.7029194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:32.7029665Z dist init r=0, world=2
2022-11-23T01:36:32.7029970Z dist init r=1, world=2
2022-11-23T01:36:32.7030334Z skip: Need at least 2 CUDA devices (1.872s)
2022-11-23T01:36:32.7030764Z   test_device_id (__main__.TestFSDPInitialization)
2022-11-23T01:36:32.7031362Z Tests passing a ``device_id``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26653
2022-11-23T01:36:32.7032010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26654
2022-11-23T01:36:32.7032855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7033381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7034143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7034782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7035377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:32.7036254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7036829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7037629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7038225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7038711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:32.7039516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7040359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7040964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:32.7041506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:32.7041924Z dist init r=0, world=2
2022-11-23T01:36:32.7042317Z dist init r=1, world=2
2022-11-23T01:36:32.7042632Z skip: Need at least 2 CUDA devices (1.612s)
2022-11-23T01:36:32.7043055Z   test_materialize_meta_module (__main__.TestFSDPInitialization)
2022-11-23T01:36:32.7043835Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26681
2022-11-23T01:36:32.7044448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26682
2022-11-23T01:36:32.7045131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7045810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7046551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7047130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7047742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:32.7048541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7049137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7049940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7050546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7051155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:32.7052087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7053044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7053752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:32.7054370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:32.7054779Z dist init r=1, world=2
2022-11-23T01:36:32.7055198Z dist init r=0, world=2
2022-11-23T01:36:32.7055546Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:36:32.7055982Z   test_sync_module_states (__main__.TestFSDPInitialization)
2022-11-23T01:36:32.7056548Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26709
2022-11-23T01:36:32.7057162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26710
2022-11-23T01:36:32.7057938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7058478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7059115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7059648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7060171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:32.7060918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7061428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7062099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7062639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7063149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:32.7064051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7064932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7065598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:32.7066156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:32.7066534Z dist init r=0, world=2
2022-11-23T01:36:32.7066800Z dist init r=1, world=2
2022-11-23T01:36:32.7067029Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:36:32.7067346Z   test_training (__main__.TestFSDPRuntime)
2022-11-23T01:36:32.7067690Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26737
2022-11-23T01:36:32.7068071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26738
2022-11-23T01:36:32.7068531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7068862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7069291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7069622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7069958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:32.7070427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:32.7070755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:32.7071168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:32.7071514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:32.7071846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:32.7072324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7072837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:36:32.7073224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:32.7073571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:32.7073819Z dist init r=0, world=2
2022-11-23T01:36:32.7073996Z dist init r=1, world=2
2022-11-23T01:36:32.7074200Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:36:32.7074337Z 
2022-11-23T01:36:32.7074524Z ----------------------------------------------------------------------
2022-11-23T01:36:32.7074765Z Ran 5 tests in 8.324s
2022-11-23T01:36:32.7074882Z 
2022-11-23T01:36:32.7074953Z OK (skipped=5)
2022-11-23T01:36:32.7075057Z 
2022-11-23T01:36:32.7075139Z Generating XML reports...
2022-11-23T01:36:32.7075600Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123013624.xml
2022-11-23T01:36:32.7076197Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123013624.xml
2022-11-23T01:36:32.7076446Z 
2022-11-23T01:36:32.7076772Z ##[endgroup]
2022-11-23T01:36:32.7077230Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_iy_0ktaz)
2022-11-23T01:36:32.7077495Z 
2022-11-23T01:36:32.7077710Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-11-23 01:36:32.701815]
2022-11-23T01:36:32.7078292Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:32.702277]
2022-11-23T01:36:43.3385808Z 
2022-11-23T01:36:43.3386624Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T01:36:43.3387802Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_faj91szt)
2022-11-23T01:36:43.3388871Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-11-23T01:36:43.3389297Z 
2022-11-23T01:36:43.3389373Z Running tests...
2022-11-23T01:36:43.3389675Z ----------------------------------------------------------------------
2022-11-23T01:36:43.3390035Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules)
2022-11-23T01:36:43.3390462Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26776
2022-11-23T01:36:43.3390847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26777
2022-11-23T01:36:43.3391167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26778
2022-11-23T01:36:43.3391489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26779
2022-11-23T01:36:43.3391955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3392275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3392712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3393060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3393400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:43.3393858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3394185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3394612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3394953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3395273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:43.3395743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3396067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3396477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3396826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3397161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:43.3397656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3398166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3398662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3398992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3399417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3399747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3400151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:43.3400655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3401162Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3401545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:43.3401897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:43.3402276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:43.3402606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:43.3402862Z dist init r=1, world=4
2022-11-23T01:36:43.3403041Z dist init r=2, world=4
2022-11-23T01:36:43.3403206Z dist init r=0, world=4
2022-11-23T01:36:43.3403385Z dist init r=3, world=4
2022-11-23T01:36:43.3403591Z skip: Need at least 2 CUDA devices (2.079s)
2022-11-23T01:36:43.3403897Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules)
2022-11-23T01:36:43.3404309Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26832
2022-11-23T01:36:43.3404688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26833
2022-11-23T01:36:43.3405016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26834
2022-11-23T01:36:43.3405329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26835
2022-11-23T01:36:43.3405779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3406109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3406536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3406868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3407293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:43.3407765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3408078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3408506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3408851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3409186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:43.3409637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3409962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3410392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3410722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3411054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:43.3411551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3412046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3412358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3412835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3413177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3413510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:43.3413989Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3414508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3415290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3415678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:43.3416017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:43.3416545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:43.3417015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:43.3417319Z dist init r=0, world=4
2022-11-23T01:36:43.3417593Z dist init r=3, world=4
2022-11-23T01:36:43.3417759Z dist init r=1, world=4
2022-11-23T01:36:43.3417943Z dist init r=2, world=4
2022-11-23T01:36:43.3418197Z skip: Need at least 2 CUDA devices (1.819s)
2022-11-23T01:36:43.3418455Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-11-23T01:36:43.3418907Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26888
2022-11-23T01:36:43.3419305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26889
2022-11-23T01:36:43.3419682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26890
2022-11-23T01:36:43.3419998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26891
2022-11-23T01:36:43.3420516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3420844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3421312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3421661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3422048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:43.3422524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3422881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3423317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3423660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3423981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:43.3424531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3425028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3425354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3425773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3426121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3426589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:43.3427094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3427576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3427904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3428328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3428655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3429029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:43.3429521Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3430086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3430464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:43.3430816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:43.3431156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:43.3431502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:43.3431746Z dist init r=1, world=4
2022-11-23T01:36:43.3431930Z dist init r=2, world=4
2022-11-23T01:36:43.3432106Z dist init r=0, world=4
2022-11-23T01:36:43.3432266Z dist init r=3, world=4
2022-11-23T01:36:43.3432470Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:36:43.3432737Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-11-23T01:36:43.3433111Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26944
2022-11-23T01:36:43.3433508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26945
2022-11-23T01:36:43.3433833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26946
2022-11-23T01:36:43.3434157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26947
2022-11-23T01:36:43.3434596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3434927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3435357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3435691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3436027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:43.3436494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3436816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3437287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3437630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3437961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:43.3438414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3438734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3439159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3439532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3439851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:43.3440345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3440833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3441156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3441596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3441944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3442273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:43.3442754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3443268Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3443778Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3444159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:43.3444492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:43.3444842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:43.3445178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:43.3445437Z dist init r=0, world=4
2022-11-23T01:36:43.3445606Z dist init r=2, world=4
2022-11-23T01:36:43.3445785Z dist init r=1, world=4
2022-11-23T01:36:43.3445956Z dist init r=3, world=4
2022-11-23T01:36:43.3446147Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:36:43.3446420Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-11-23T01:36:43.3446916Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27000
2022-11-23T01:36:43.3447370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27001
2022-11-23T01:36:43.3447698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27002
2022-11-23T01:36:43.3448020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27003
2022-11-23T01:36:43.3448473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3448792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3449221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3449569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3449891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:43.3450405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3450736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3451160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3451491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3451824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:43.3452330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3452658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3453071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3453414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3453745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:43.3454224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3454755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:43.3455233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:43.3455675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:43.3456006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:43.3456340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:43.3456832Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3457349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3457852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:43.3458239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:43.3458587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:43.3458920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:43.3459260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:43.3459517Z dist init r=3, world=4
2022-11-23T01:36:43.3459698Z dist init r=1, world=4
2022-11-23T01:36:43.3459862Z dist init r=2, world=4
2022-11-23T01:36:43.3460038Z dist init r=0, world=4
2022-11-23T01:36:43.3460244Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:36:43.3460370Z 
2022-11-23T01:36:43.3460569Z ----------------------------------------------------------------------
2022-11-23T01:36:43.3460813Z Ran 5 tests in 9.057s
2022-11-23T01:36:43.3460928Z 
2022-11-23T01:36:43.3460999Z OK (skipped=5)
2022-11-23T01:36:43.3461106Z 
2022-11-23T01:36:43.3461176Z Generating XML reports...
2022-11-23T01:36:43.3461659Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123013633.xml
2022-11-23T01:36:43.3461944Z 
2022-11-23T01:36:43.3462265Z ##[endgroup]
2022-11-23T01:36:43.3462734Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_faj91szt)
2022-11-23T01:36:43.3463015Z 
2022-11-23T01:36:43.3463228Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-11-23 01:36:43.339150]
2022-11-23T01:36:43.3463770Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:43.339592]
2022-11-23T01:36:54.1864569Z 
2022-11-23T01:36:54.1865166Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2022-11-23T01:36:54.1866426Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_8rsdra31)
2022-11-23T01:36:54.1867893Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2022-11-23T01:36:54.1868120Z 
2022-11-23T01:36:54.1868199Z Running tests...
2022-11-23T01:36:54.1868499Z ----------------------------------------------------------------------
2022-11-23T01:36:54.1868818Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2022-11-23T01:36:54.1869203Z Tests checkpointing for TP + FSDP integration. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27067
2022-11-23T01:36:54.1869591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27068
2022-11-23T01:36:54.1869974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27069
2022-11-23T01:36:54.1870296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27070
2022-11-23T01:36:54.1870773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1871097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1871531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1871877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1872215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:54.1872671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1873002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1873430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1873773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1874094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:54.1874592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1875092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1875403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1875832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1876177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1876518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:54.1876971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1877300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1877727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1878071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1878390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:54.1878883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1879405Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1879913Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1880303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:54.1880686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:54.1881032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:54.1881362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:54.1881615Z dist init r=2, world=4
2022-11-23T01:36:54.1881798Z dist init r=1, world=4
2022-11-23T01:36:54.1881961Z dist init r=0, world=4
2022-11-23T01:36:54.1882138Z dist init r=3, world=4
2022-11-23T01:36:54.1882345Z skip: Need at least 4 CUDA devices (2.180s)
2022-11-23T01:36:54.1882677Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T01:36:54.1883280Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27123
2022-11-23T01:36:54.1883681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27124
2022-11-23T01:36:54.1884014Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27125
2022-11-23T01:36:54.1884328Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27126
2022-11-23T01:36:54.1884779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1885111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1885539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1885875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1886209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:54.1886680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1886996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1887507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1887856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1888191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:54.1888644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1888980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1889411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1889756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1890072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:54.1890550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1890876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1891290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1891636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1891969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:54.1892466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1892980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1893541Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1894060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1894444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:54.1894783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:54.1895331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:54.1895769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:54.1896166Z dist init r=0, world=4
2022-11-23T01:36:54.1896350Z dist init r=3, world=4
2022-11-23T01:36:54.1896573Z dist init r=2, world=4
2022-11-23T01:36:54.1896813Z dist init r=1, world=4
2022-11-23T01:36:54.1897147Z skip: Need at least 4 CUDA devices (1.821s)
2022-11-23T01:36:54.1897724Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T01:36:54.1898683Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27179
2022-11-23T01:36:54.1899075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27180
2022-11-23T01:36:54.1899409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27181
2022-11-23T01:36:54.1899738Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27182
2022-11-23T01:36:54.1900188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1900521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1900952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1901301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1901629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:54.1902099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1902429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1902857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1903260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1903585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:54.1904054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1904387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1904802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1905150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1905487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:54.1905954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1906268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1906694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1907028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1907367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:54.1907933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1908455Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1908960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1909471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1909895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:54.1910243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:54.1910574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:54.1910912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:54.1911172Z dist init r=3, world=4
2022-11-23T01:36:54.1911339Z dist init r=1, world=4
2022-11-23T01:36:54.1911516Z dist init r=0, world=4
2022-11-23T01:36:54.1911692Z dist init r=2, world=4
2022-11-23T01:36:54.1911886Z skip: Need at least 4 CUDA devices (1.720s)
2022-11-23T01:36:54.1912229Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T01:36:54.1912796Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27235
2022-11-23T01:36:54.1913197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27236
2022-11-23T01:36:54.1913511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27237
2022-11-23T01:36:54.1913834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27238
2022-11-23T01:36:54.1914283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1914611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1915025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1915370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1915705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:54.1916162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1916486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1916910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1917252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1917570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:54.1918033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1918356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1918782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1919107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1919442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:54.1919906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1920253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1920683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1921024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1921355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:54.1921838Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1922357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1922906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1923412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1923782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:54.1924131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:54.1924469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:54.1924795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:54.1925052Z dist init r=0, world=4
2022-11-23T01:36:54.1925233Z dist init r=2, world=4
2022-11-23T01:36:54.1925398Z dist init r=1, world=4
2022-11-23T01:36:54.1925575Z dist init r=3, world=4
2022-11-23T01:36:54.1925778Z skip: Need at least 4 CUDA devices (1.820s)
2022-11-23T01:36:54.1926116Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T01:36:54.1926668Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27291
2022-11-23T01:36:54.1927171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27292
2022-11-23T01:36:54.1927508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27293
2022-11-23T01:36:54.1927817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27294
2022-11-23T01:36:54.1928268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1928594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1929022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1929351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1929683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:36:54.1930150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1930474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1930885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1931227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1931554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:36:54.1932005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1932327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1932752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1933132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1933450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:36:54.1933918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:36:54.1934242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:36:54.1934653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:36:54.1935120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:36:54.1935514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:36:54.1936009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1936514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1937024Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1937540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:36:54.1937928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:36:54.1938261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:36:54.1938601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:36:54.1938939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:36:54.1939196Z dist init r=0, world=4
2022-11-23T01:36:54.1939363Z dist init r=2, world=4
2022-11-23T01:36:54.1939539Z dist init r=1, world=4
2022-11-23T01:36:54.1939714Z dist init r=3, world=4
2022-11-23T01:36:54.1939905Z skip: Need at least 4 CUDA devices (1.720s)
2022-11-23T01:36:54.1940041Z 
2022-11-23T01:36:54.1940240Z ----------------------------------------------------------------------
2022-11-23T01:36:54.1940481Z Ran 5 tests in 9.263s
2022-11-23T01:36:54.1940593Z 
2022-11-23T01:36:54.1940653Z OK (skipped=5)
2022-11-23T01:36:54.1940758Z 
2022-11-23T01:36:54.1940842Z Generating XML reports...
2022-11-23T01:36:54.1941322Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123013644.xml
2022-11-23T01:36:54.1941603Z 
2022-11-23T01:36:54.1941899Z ##[endgroup]
2022-11-23T01:36:54.1942378Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_8rsdra31)
2022-11-23T01:36:54.1942656Z 
2022-11-23T01:36:54.1942877Z Running distributed/elastic/timer/local_timer_example ... [2022-11-23 01:36:54.186755]
2022-11-23T01:36:54.1943421Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:36:54.187095]
2022-11-23T01:37:05.7479720Z 
2022-11-23T01:37:05.7480348Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2022-11-23T01:37:05.7481526Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_g7ncz9e_)
2022-11-23T01:37:05.7482720Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-11-23T01:37:05.7483149Z 
2022-11-23T01:37:05.7483280Z Running tests...
2022-11-23T01:37:05.7483588Z ----------------------------------------------------------------------
2022-11-23T01:37:05.7484321Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-11-23 01:36:55,659 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T01:37:05.7484744Z [INFO] 2022-11-23 01:36:55,659 api: Starting watchdog thread...
2022-11-23T01:37:05.7485164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7485484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7485917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7486266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7486685Z [INFO] 2022-11-23 01:36:57,743 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7487196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7487527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7487967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7488299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7488796Z [INFO] 2022-11-23 01:36:57,753 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7489282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7489611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7490115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7490494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7490878Z [INFO] 2022-11-23 01:36:57,761 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7491353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7491685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7492169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7492515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7492914Z [INFO] 2022-11-23 01:36:57,762 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7493355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7493740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7494155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7494553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7495147Z [INFO] 2022-11-23 01:36:57,774 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7495592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7495952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7496381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7496777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7497157Z [INFO] 2022-11-23 01:36:57,786 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7497745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7498170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7498766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7499101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7499463Z [INFO] 2022-11-23 01:36:57,880 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7499895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7500222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7500639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7501033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7501401Z [INFO] 2022-11-23 01:36:57,882 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7501816Z [INFO] 2022-11-23 01:36:58,891 api: Reaping worker_id=[27361]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7502201Z [INFO] 2022-11-23 01:36:58,892 api: Successfully reaped worker=[27361]
2022-11-23T01:37:05.7502621Z [INFO] 2022-11-23 01:36:58,892 api: Reaping worker_id=[27363]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7502998Z [INFO] 2022-11-23 01:36:58,892 api: Successfully reaped worker=[27363]
2022-11-23T01:37:05.7503407Z [INFO] 2022-11-23 01:36:58,962 api: Reaping worker_id=[27365]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7503782Z [INFO] 2022-11-23 01:36:58,962 api: Successfully reaped worker=[27365]
2022-11-23T01:37:05.7504196Z [INFO] 2022-11-23 01:36:58,983 api: Reaping worker_id=[27359]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7504568Z [INFO] 2022-11-23 01:36:58,983 api: Successfully reaped worker=[27359]
2022-11-23T01:37:05.7504873Z [INFO] 2022-11-23 01:36:58,991 api: Stopping LocalTimerServer
2022-11-23T01:37:05.7505189Z [INFO] 2022-11-23 01:36:58,991 api: Stopping watchdog thread...
2022-11-23T01:37:05.7505395Z ok (3.594s)
2022-11-23T01:37:05.7505806Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-11-23 01:36:58,995 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T01:37:05.7506206Z [INFO] 2022-11-23 01:36:58,996 api: Starting watchdog thread...
2022-11-23T01:37:05.7506619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7506931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7507358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7507704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7508069Z [INFO] 2022-11-23 01:37:01,091 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7508493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7508851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7509272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7509615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7509962Z [INFO] 2022-11-23 01:37:01,101 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7510391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7510722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7511130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7511471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7511863Z [INFO] 2022-11-23 01:37:01,107 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7512296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7512608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7513029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7513369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7513714Z [INFO] 2022-11-23 01:37:01,114 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7514204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7514525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7514950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7515279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7515638Z [INFO] 2022-11-23 01:37:01,116 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7516068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7516381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7516803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7517148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7517510Z [INFO] 2022-11-23 01:37:01,127 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7517928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7518253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7518676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7519019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7519365Z [INFO] 2022-11-23 01:37:01,131 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7519790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7520116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7520528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7520872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7521227Z [INFO] 2022-11-23 01:37:01,131 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7521652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7521961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7522380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7522720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7523063Z [INFO] 2022-11-23 01:37:04,218 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7523486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7523810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7524232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7524589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7524949Z [INFO] 2022-11-23 01:37:04,225 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7525376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7525742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7526148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7526491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7526884Z [INFO] 2022-11-23 01:37:04,232 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7527424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7527736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7528166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7528510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7528856Z [INFO] 2022-11-23 01:37:04,234 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7529288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7529612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7530037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7530367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7530724Z [INFO] 2022-11-23 01:37:04,242 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7531156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7531468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7531891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7532230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7532592Z [INFO] 2022-11-23 01:37:04,252 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7533004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7533328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7533749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7534089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7534438Z [INFO] 2022-11-23 01:37:04,255 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7534990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:05.7535341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:05.7535753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:05.7536097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:05.7536456Z [INFO] 2022-11-23 01:37:04,265 api: Timer client configured to: LocalTimerClient
2022-11-23T01:37:05.7536891Z [INFO] 2022-11-23 01:37:05,362 api: Reaping worker_id=[27542]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7537256Z [INFO] 2022-11-23 01:37:05,363 api: Successfully reaped worker=[27542]
2022-11-23T01:37:05.7537743Z [INFO] 2022-11-23 01:37:05,373 api: Reaping worker_id=[27539]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7538116Z [INFO] 2022-11-23 01:37:05,373 api: Successfully reaped worker=[27539]
2022-11-23T01:37:05.7538515Z [INFO] 2022-11-23 01:37:05,373 api: Reaping worker_id=[27538]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7538893Z [INFO] 2022-11-23 01:37:05,373 api: Successfully reaped worker=[27538]
2022-11-23T01:37:05.7539303Z [INFO] 2022-11-23 01:37:05,383 api: Reaping worker_id=[27537]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7539676Z [INFO] 2022-11-23 01:37:05,383 api: Successfully reaped worker=[27537]
2022-11-23T01:37:05.7540120Z [INFO] 2022-11-23 01:37:05,383 api: Reaping worker_id=[27540]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7540488Z [INFO] 2022-11-23 01:37:05,383 api: Successfully reaped worker=[27540]
2022-11-23T01:37:05.7540900Z [INFO] 2022-11-23 01:37:05,383 api: Reaping worker_id=[27541]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7541273Z [INFO] 2022-11-23 01:37:05,383 api: Successfully reaped worker=[27541]
2022-11-23T01:37:05.7541668Z [INFO] 2022-11-23 01:37:05,394 api: Reaping worker_id=[27543]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7542041Z [INFO] 2022-11-23 01:37:05,394 api: Successfully reaped worker=[27543]
2022-11-23T01:37:05.7542363Z [INFO] 2022-11-23 01:37:05,402 api: Stopping LocalTimerServer
2022-11-23T01:37:05.7542663Z [INFO] 2022-11-23 01:37:05,402 api: Stopping watchdog thread...
2022-11-23T01:37:05.7543065Z [INFO] 2022-11-23 01:37:05,404 api: Reaping worker_id=[27536]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-11-23T01:37:05.7543474Z [INFO] 2022-11-23 01:37:05,404 local_timer: Process with pid=27536 does not exist. Skipping
2022-11-23T01:37:05.7543821Z [INFO] 2022-11-23 01:37:05,404 api: Successfully reaped worker=[27536]
2022-11-23T01:37:05.7544022Z ok (6.411s)
2022-11-23T01:37:05.7544188Z 
2022-11-23T01:37:05.7544389Z ----------------------------------------------------------------------
2022-11-23T01:37:05.7544628Z Ran 2 tests in 10.006s
2022-11-23T01:37:05.7544742Z 
2022-11-23T01:37:05.7544788Z OK
2022-11-23T01:37:05.7544878Z 
2022-11-23T01:37:05.7544961Z Generating XML reports...
2022-11-23T01:37:05.7545437Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123013655.xml
2022-11-23T01:37:05.7545718Z 
2022-11-23T01:37:05.7546004Z ##[endgroup]
2022-11-23T01:37:05.7546504Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_g7ncz9e_)
2022-11-23T01:37:05.7546796Z 
2022-11-23T01:37:05.7546984Z Running distributed/test_c10d_common ... [2022-11-23 01:37:05.748786]
2022-11-23T01:37:05.7547510Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:05.749669]
2022-11-23T01:37:38.0413175Z 
2022-11-23T01:37:38.0416480Z Expand the folded group to see the log file of distributed/test_c10d_common
2022-11-23T01:37:38.0417334Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_5z_kuxxp)
2022-11-23T01:37:38.0417746Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2022-11-23T01:37:38.0418246Z test_debug_level (__main__.CommTest)
2022-11-23T01:37:38.0419122Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2022-11-23T01:37:38.0419756Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:37:38.0420269Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:37:38.0420574Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:37:38.0420892Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-11-23T01:37:38.0421545Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2022-11-23T01:37:38.0422250Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:37:38.0422555Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:37:38.0422876Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:37:38.0423186Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-11-23T01:37:38.0423644Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>, <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]>
2022-11-23T01:37:38.0424074Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2022-11-23T01:37:38.0424331Z test_reduceop_copyable (__main__.ReduceOpTest)
2022-11-23T01:37:38.0424571Z test_reduceop_pickle (__main__.ReduceOpTest)
2022-11-23T01:37:38.0425028Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0425503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0425839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0426256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0426606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0426773Z 
2022-11-23T01:37:38.0426849Z Running tests...
2022-11-23T01:37:38.0427156Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0427508Z   test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27645
2022-11-23T01:37:38.0427874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27646
2022-11-23T01:37:38.0428328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0428654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0429068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0429415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0429739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:37:38.0430184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0430511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0430937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0431284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0431594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:38.0431847Z ok (1.783s)
2022-11-23T01:37:38.0431950Z 
2022-11-23T01:37:38.0432148Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0432373Z Ran 1 test in 1.784s
2022-11-23T01:37:38.0432484Z 
2022-11-23T01:37:38.0432546Z OK
2022-11-23T01:37:38.0432637Z 
2022-11-23T01:37:38.0432758Z Generating XML reports...
2022-11-23T01:37:38.0433170Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123013708.xml
2022-11-23T01:37:38.0433618Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0434080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0434534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0434953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0435347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0435510Z 
2022-11-23T01:37:38.0435585Z Running tests...
2022-11-23T01:37:38.0435888Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0436203Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.259s)
2022-11-23T01:37:38.0436388Z 
2022-11-23T01:37:38.0436579Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0436814Z Ran 1 test in 0.260s
2022-11-23T01:37:38.0436924Z 
2022-11-23T01:37:38.0436972Z OK
2022-11-23T01:37:38.0437061Z 
2022-11-23T01:37:38.0437144Z Generating XML reports...
2022-11-23T01:37:38.0437615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013711.xml
2022-11-23T01:37:38.0438119Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0438567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0438894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0439323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0439654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0439817Z 
2022-11-23T01:37:38.0439889Z Running tests...
2022-11-23T01:37:38.0440186Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0440512Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.257s)
2022-11-23T01:37:38.0440698Z 
2022-11-23T01:37:38.0440877Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0441111Z Ran 1 test in 0.257s
2022-11-23T01:37:38.0441224Z 
2022-11-23T01:37:38.0441285Z OK
2022-11-23T01:37:38.0441375Z 
2022-11-23T01:37:38.0441445Z Generating XML reports...
2022-11-23T01:37:38.0441907Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013713.xml
2022-11-23T01:37:38.0442410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0442873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0443188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0443617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0443960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0444124Z 
2022-11-23T01:37:38.0444197Z Running tests...
2022-11-23T01:37:38.0444480Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0444806Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.255s)
2022-11-23T01:37:38.0444992Z 
2022-11-23T01:37:38.0445185Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0445407Z Ran 1 test in 0.255s
2022-11-23T01:37:38.0445518Z 
2022-11-23T01:37:38.0445578Z OK
2022-11-23T01:37:38.0445702Z 
2022-11-23T01:37:38.0445789Z Generating XML reports...
2022-11-23T01:37:38.0446255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013715.xml
2022-11-23T01:37:38.0446742Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0447322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0447656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0448117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0448465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0448630Z 
2022-11-23T01:37:38.0448704Z Running tests...
2022-11-23T01:37:38.0449004Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0449317Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.254s)
2022-11-23T01:37:38.0449508Z 
2022-11-23T01:37:38.0449699Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0449936Z Ran 1 test in 0.254s
2022-11-23T01:37:38.0450049Z 
2022-11-23T01:37:38.0450097Z OK
2022-11-23T01:37:38.0450189Z 
2022-11-23T01:37:38.0450274Z Generating XML reports...
2022-11-23T01:37:38.0450744Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013716.xml
2022-11-23T01:37:38.0451248Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0451691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0452017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0460677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0461035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0461205Z 
2022-11-23T01:37:38.0461280Z Running tests...
2022-11-23T01:37:38.0461591Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0461999Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27718
2022-11-23T01:37:38.0462422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27719
2022-11-23T01:37:38.0462762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27720
2022-11-23T01:37:38.0463094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27721
2022-11-23T01:37:38.0463539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0463877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0464311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0464660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0464971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:38.0465429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0465757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0466170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0466514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0466950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:37:38.0467415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0467729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0468159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0468586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0468908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:37:38.0469419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0469731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0470166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0470512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0470832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:37:38.0471073Z ok (1.978s)
2022-11-23T01:37:38.0471175Z 
2022-11-23T01:37:38.0471377Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0471619Z Ran 1 test in 1.978s
2022-11-23T01:37:38.0471730Z 
2022-11-23T01:37:38.0471779Z OK
2022-11-23T01:37:38.0471870Z 
2022-11-23T01:37:38.0471956Z Generating XML reports...
2022-11-23T01:37:38.0472449Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013718.xml
2022-11-23T01:37:38.0472969Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0473418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0473750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0474181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0474517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0474683Z 
2022-11-23T01:37:38.0474756Z Running tests...
2022-11-23T01:37:38.0475057Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0475469Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27773
2022-11-23T01:37:38.0475870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27774
2022-11-23T01:37:38.0476203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27775
2022-11-23T01:37:38.0476533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27776
2022-11-23T01:37:38.0476987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0477305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0477732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0478078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0478388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:37:38.0478873Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T01:37:38.0479357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0479685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0480134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0480481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0480803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:37:38.0481267Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-11-23T01:37:38.0481744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0482106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0482532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0482862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0483185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:37:38.0483643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0483971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0484387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0484729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0485047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:38.0485400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:37:38.0485771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:37:38.0486140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:37:38.0486505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:38.0487066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0487597Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0488116Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0488631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0488906Z ok (2.977s)
2022-11-23T01:37:38.0489008Z 
2022-11-23T01:37:38.0489207Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0489452Z Ran 1 test in 2.977s
2022-11-23T01:37:38.0489565Z 
2022-11-23T01:37:38.0489614Z OK
2022-11-23T01:37:38.0489705Z 
2022-11-23T01:37:38.0489790Z Generating XML reports...
2022-11-23T01:37:38.0490276Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013722.xml
2022-11-23T01:37:38.0490789Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0491242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0491571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0492001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0492347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0492498Z 
2022-11-23T01:37:38.0492573Z Running tests...
2022-11-23T01:37:38.0492916Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0493331Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27833
2022-11-23T01:37:38.0493735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27834
2022-11-23T01:37:38.0494067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27835
2022-11-23T01:37:38.0494397Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27836
2022-11-23T01:37:38.0494979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0495372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0495806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0496156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0496465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:37:38.0496924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0497251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0497676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0498007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0498333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:37:38.0498791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0499102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0499529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0499873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0500194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:38.0500636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0500966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0501389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0501735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0502042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:37:38.0502294Z ok (1.975s)
2022-11-23T01:37:38.0502397Z 
2022-11-23T01:37:38.0502601Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0502828Z Ran 1 test in 1.975s
2022-11-23T01:37:38.0502940Z 
2022-11-23T01:37:38.0503000Z OK
2022-11-23T01:37:38.0503094Z 
2022-11-23T01:37:38.0503178Z Generating XML reports...
2022-11-23T01:37:38.0503648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013726.xml
2022-11-23T01:37:38.0504165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0504628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0504955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0505371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0505782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0505945Z 
2022-11-23T01:37:38.0506019Z Running tests...
2022-11-23T01:37:38.0506321Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0506713Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27888
2022-11-23T01:37:38.0507125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27889
2022-11-23T01:37:38.0507461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27890
2022-11-23T01:37:38.0507806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27891
2022-11-23T01:37:38.0508253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0508582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0509010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0509343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0509664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:38.0510120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0510436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0510862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0511206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0511528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:37:38.0511880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:37:38.0512348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0512674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0513100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0513433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0513755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:37:38.0514119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:37:38.0514576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0514903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0515331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0515673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0515985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:37:38.0516345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:37:38.0516712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:38.0517212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0517723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0518271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0518788Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:37:38.0519059Z ok (1.976s)
2022-11-23T01:37:38.0519165Z 
2022-11-23T01:37:38.0519363Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0519603Z Ran 1 test in 1.976s
2022-11-23T01:37:38.0519715Z 
2022-11-23T01:37:38.0519775Z OK
2022-11-23T01:37:38.0519852Z 
2022-11-23T01:37:38.0519937Z Generating XML reports...
2022-11-23T01:37:38.0520422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013730.xml
2022-11-23T01:37:38.0520974Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0521425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0521759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0522187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0522530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0522692Z 
2022-11-23T01:37:38.0522751Z Running tests...
2022-11-23T01:37:38.0523049Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0523349Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... ok (0.255s)
2022-11-23T01:37:38.0523514Z 
2022-11-23T01:37:38.0523711Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0523934Z Ran 1 test in 0.255s
2022-11-23T01:37:38.0524046Z 
2022-11-23T01:37:38.0524107Z OK
2022-11-23T01:37:38.0524198Z 
2022-11-23T01:37:38.0524281Z Generating XML reports...
2022-11-23T01:37:38.0524689Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013733.xml
2022-11-23T01:37:38.0525154Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0525613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0525931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0526360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0526700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0526944Z 
2022-11-23T01:37:38.0527019Z Running tests...
2022-11-23T01:37:38.0527312Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0527608Z   test_reduceop_copyable (__main__.ReduceOpTest) ... ok (0.256s)
2022-11-23T01:37:38.0527763Z 
2022-11-23T01:37:38.0527958Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0528185Z Ran 1 test in 0.256s
2022-11-23T01:37:38.0528300Z 
2022-11-23T01:37:38.0528361Z OK
2022-11-23T01:37:38.0528452Z 
2022-11-23T01:37:38.0528535Z Generating XML reports...
2022-11-23T01:37:38.0528955Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013735.xml
2022-11-23T01:37:38.0529407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-11-23T01:37:38.0529863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:38.0530192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:38.0530619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:38.0530946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:38.0531107Z 
2022-11-23T01:37:38.0531181Z Running tests...
2022-11-23T01:37:38.0531512Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0531789Z   test_reduceop_pickle (__main__.ReduceOpTest) ... ok (0.258s)
2022-11-23T01:37:38.0531941Z 
2022-11-23T01:37:38.0532134Z ----------------------------------------------------------------------
2022-11-23T01:37:38.0532368Z Ran 1 test in 0.258s
2022-11-23T01:37:38.0532482Z 
2022-11-23T01:37:38.0532543Z OK
2022-11-23T01:37:38.0532620Z 
2022-11-23T01:37:38.0532703Z Generating XML reports...
2022-11-23T01:37:38.0533118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013737.xml
2022-11-23T01:37:38.0533390Z 
2022-11-23T01:37:38.0533696Z ##[endgroup]
2022-11-23T01:37:38.0534120Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_5z_kuxxp)
2022-11-23T01:37:38.0534364Z 
2022-11-23T01:37:38.0534578Z Running distributed/fsdp/test_fsdp_flatten_params ... [2022-11-23 01:37:38.042001]
2022-11-23T01:37:38.0535261Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:38.042370]
2022-11-23T01:37:53.4836233Z 
2022-11-23T01:37:53.4836779Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_flatten_params
2022-11-23T01:37:53.4837749Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_7dlw6wkx)
2022-11-23T01:37:53.4838699Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params
2022-11-23T01:37:53.4838916Z 
2022-11-23T01:37:53.4838992Z Running tests...
2022-11-23T01:37:53.4839335Z ----------------------------------------------------------------------
2022-11-23T01:37:53.4856250Z   test_empty_module (__main__.TestFlattenParams)
2022-11-23T01:37:53.4857334Z Tests flattening an empty module (i.e. one without any parameters). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27978
2022-11-23T01:37:53.4858387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4858733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4859212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4859567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4859964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4860469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4860904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4895695Z dist init r=0, world=1
2022-11-23T01:37:53.4896122Z skip: Need at least 1 CUDA device (1.775s)
2022-11-23T01:37:53.4896576Z   test_flat_param_shard_metadata (__main__.TestFlattenParams)
2022-11-23T01:37:53.4897051Z Tests that ``FlatParameter`` shard metadata are computed as expected. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27992
2022-11-23T01:37:53.4897630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4897967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4898415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4898769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4899094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4900032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4900695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4901134Z dist init r=0, world=1
2022-11-23T01:37:53.4901402Z ok (1.511s)
2022-11-23T01:37:53.4901781Z   test_flatten_nothing (__main__.TestFlattenParams)
2022-11-23T01:37:53.4902483Z Tests that constructing a ``FlatParamHandle`` with no parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28006
2022-11-23T01:37:53.4903555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4904008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4904448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4905041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4905614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4906387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4907018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4907470Z dist init r=0, world=1
2022-11-23T01:37:53.4907757Z ok (1.508s)
2022-11-23T01:37:53.4908234Z   test_numel_with_shared_params (__main__.TestFlattenParams)
2022-11-23T01:37:53.4908946Z Tests that numel is preserved after flattening when there are shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28020
2022-11-23T01:37:53.4909731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4910052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4910486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4910832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4911156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4911663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4912058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4912325Z dist init r=0, world=1
2022-11-23T01:37:53.4912490Z ok (1.508s)
2022-11-23T01:37:53.4912728Z   test_numel_without_shared_params (__main__.TestFlattenParams)
2022-11-23T01:37:53.4913127Z Tests that numel is preserved after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28034
2022-11-23T01:37:53.4913643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4913976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4914409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4914762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4915091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4915594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4915989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4916239Z dist init r=0, world=1
2022-11-23T01:37:53.4916415Z ok (1.508s)
2022-11-23T01:37:53.4916716Z   test_output_with_shared_params (__main__.TestFlattenParams)
2022-11-23T01:37:53.4917115Z Tests a forward pass after flattening when there are shared parameters ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28048
2022-11-23T01:37:53.4917636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4917969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4918398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4918766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4919104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4919611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4920007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4920255Z dist init r=0, world=1
2022-11-23T01:37:53.4920467Z skip: Need at least 1 CUDA device (1.508s)
2022-11-23T01:37:53.4920734Z   test_output_without_shared_params (__main__.TestFlattenParams)
2022-11-23T01:37:53.4921111Z Tests a forward pass after flattening when there are no shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28062
2022-11-23T01:37:53.4921632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4921963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4922393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4922730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4923075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4923579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4923970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4924219Z dist init r=0, world=1
2022-11-23T01:37:53.4924430Z skip: Need at least 1 CUDA device (1.508s)
2022-11-23T01:37:53.4924683Z   test_partial_flattening (__main__.TestFlattenParams)
2022-11-23T01:37:53.4925032Z Tests flattening some submodules but not others. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28076
2022-11-23T01:37:53.4925544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4925870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4926308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4926650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4927081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4927591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4927971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4928239Z dist init r=0, world=1
2022-11-23T01:37:53.4928447Z skip: Need at least 1 CUDA device (1.508s)
2022-11-23T01:37:53.4928720Z   test_pnorm_after_step_with_shared_params (__main__.TestFlattenParams)
2022-11-23T01:37:53.4929145Z Tests for parameter Frobenius norm parity after an optimizer step when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28090
2022-11-23T01:37:53.4929680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:37:53.4930011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:37:53.4930430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:37:53.4930776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:37:53.4931117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:37:53.4931655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T01:37:53.4932033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:37:53.4932296Z dist init r=0, world=1
2022-11-23T01:37:53.4932517Z skip: Need at least 1 CUDA device (1.508s)
2022-11-23T01:37:53.4932653Z 
2022-11-23T01:37:53.4932859Z ----------------------------------------------------------------------
2022-11-23T01:37:53.4933090Z Ran 9 tests in 13.846s
2022-11-23T01:37:53.4933204Z 
2022-11-23T01:37:53.4933278Z OK (skipped=5)
2022-11-23T01:37:53.4933386Z 
2022-11-23T01:37:53.4933471Z Generating XML reports...
2022-11-23T01:37:53.4933927Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123013739.xml
2022-11-23T01:37:53.4934200Z 
2022-11-23T01:37:53.4934620Z ##[endgroup]
2022-11-23T01:37:53.4935243Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_flatten_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_flatten_params_7dlw6wkx)
2022-11-23T01:37:53.4935520Z 
2022-11-23T01:37:53.4935716Z Running distributed/algorithms/test_join ... [2022-11-23 01:37:53.483995]
2022-11-23T01:37:53.4936221Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:37:53.484346]
2022-11-23T01:38:09.7264207Z 
2022-11-23T01:38:09.7269137Z Expand the folded group to see the log file of distributed/algorithms/test_join
2022-11-23T01:38:09.7270274Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_68g_ty75)
2022-11-23T01:38:09.7271182Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-11-23T01:38:09.7271395Z 
2022-11-23T01:38:09.7271455Z Running tests...
2022-11-23T01:38:09.7271962Z ----------------------------------------------------------------------
2022-11-23T01:38:09.7272364Z   test_join_kwargs (__main__.TestJoin)
2022-11-23T01:38:09.7272918Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28115
2022-11-23T01:38:09.7273305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28116
2022-11-23T01:38:09.7273784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7274163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7274665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7275050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7275436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7275909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7276223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7276651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7277196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7277512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7277876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7278244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7278741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7279320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7279609Z ok (1.869s)
2022-11-23T01:38:09.7279818Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-11-23T01:38:09.7280173Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28143
2022-11-23T01:38:09.7280542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28144
2022-11-23T01:38:09.7280995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7281321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7281736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7282080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7282404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7282866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7283178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7283605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7283949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7284257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7284619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7284985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7285478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7285983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7286269Z ok (1.513s)
2022-11-23T01:38:09.7286472Z   test_multiple_joinables (__main__.TestJoin)
2022-11-23T01:38:09.7287026Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28171
2022-11-23T01:38:09.7287410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28172
2022-11-23T01:38:09.7287859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7288190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7288606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7288952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7289275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7289774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7290092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7290517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7290897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7291288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7291656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7292270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7292819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7293396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7293672Z ok (1.613s)
2022-11-23T01:38:09.7293882Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-11-23T01:38:09.7294309Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28199
2022-11-23T01:38:09.7294692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28200
2022-11-23T01:38:09.7295309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7295745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7296600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7297180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7297718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7298476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7299086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7299870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7300503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7300865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7301215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7301582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7302085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7302664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7302939Z ok (1.613s)
2022-11-23T01:38:09.7303134Z   test_single_joinable (__main__.TestJoin)
2022-11-23T01:38:09.7303825Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28227
2022-11-23T01:38:09.7304452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28228
2022-11-23T01:38:09.7305159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7305667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7306279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7306878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7307354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7308046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7308577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7309263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7309819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7310408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7310993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7311598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7312403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7313179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7313580Z ok (1.613s)
2022-11-23T01:38:09.7313883Z   test_single_joinable_disable (__main__.TestJoin)
2022-11-23T01:38:09.7314410Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28255
2022-11-23T01:38:09.7314971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28256
2022-11-23T01:38:09.7315634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7316101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7316779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7317319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7317865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7318611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7319108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7319788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7320341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7320847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7321429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7321996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7322838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7323711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7324146Z ok (1.613s)
2022-11-23T01:38:09.7324458Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-11-23T01:38:09.7325002Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28283
2022-11-23T01:38:09.7325588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28284
2022-11-23T01:38:09.7326334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7327004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7327658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7328239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7328726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7329477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7329980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7330666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7331165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7331637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7332154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7332692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7333495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7334316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7334792Z ok (1.613s)
2022-11-23T01:38:09.7335222Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-11-23T01:38:09.7335992Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28311
2022-11-23T01:38:09.7336595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28312
2022-11-23T01:38:09.7337427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7337981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7338751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7339387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7339914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7340715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7341317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7342049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7342630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7343170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7343777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7344383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7345249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7346159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7346684Z ok (1.613s)
2022-11-23T01:38:09.7347036Z   test_single_joinable_throw (__main__.TestJoin)
2022-11-23T01:38:09.7347635Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28339
2022-11-23T01:38:09.7348393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28340
2022-11-23T01:38:09.7349188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7349755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7350523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7351129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7351683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:09.7352636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:09.7353238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:09.7354014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:09.7354642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:09.7355236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:09.7355907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:09.7356567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:09.7357452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7358361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:38:09.7358845Z ok (1.613s)
2022-11-23T01:38:09.7359010Z 
2022-11-23T01:38:09.7359369Z ----------------------------------------------------------------------
2022-11-23T01:38:09.7359799Z Ran 9 tests in 14.676s
2022-11-23T01:38:09.7360002Z 
2022-11-23T01:38:09.7360108Z OK
2022-11-23T01:38:09.7360271Z 
2022-11-23T01:38:09.7360410Z Generating XML reports...
2022-11-23T01:38:09.7361188Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123013754.xml
2022-11-23T01:38:09.7361611Z 
2022-11-23T01:38:09.7362080Z ##[endgroup]
2022-11-23T01:38:09.7362869Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_68g_ty75)
2022-11-23T01:38:09.7363313Z 
2022-11-23T01:38:09.7363664Z Running distributed/fsdp/test_fsdp_comm ... [2022-11-23 01:38:09.726881]
2022-11-23T01:38:09.7364545Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:09.727364]
2022-11-23T01:38:25.8448500Z 
2022-11-23T01:38:25.8449433Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2022-11-23T01:38:25.8460143Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_onzwi1iw)
2022-11-23T01:38:25.8461459Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-11-23T01:38:25.8461867Z 
2022-11-23T01:38:25.8462024Z Running tests...
2022-11-23T01:38:25.8462541Z ----------------------------------------------------------------------
2022-11-23T01:38:25.8463117Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T01:38:25.8464070Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28378
2022-11-23T01:38:25.8464788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28379
2022-11-23T01:38:25.8465393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28380
2022-11-23T01:38:25.8466258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28381
2022-11-23T01:38:25.8467117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8467684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8468482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8469088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8469764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8470654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8471255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8472069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8472716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8473302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8474181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8474720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8475395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8475953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8476484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8477228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8477758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8478468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8479034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8479563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8480396Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8481212Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8482095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8482959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8483602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8484179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8484776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8485429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8485900Z dist init r=1, world=4
2022-11-23T01:38:25.8486222Z dist init r=2, world=4
2022-11-23T01:38:25.8486514Z dist init r=3, world=4
2022-11-23T01:38:25.8486919Z dist init r=0, world=4
2022-11-23T01:38:25.8487295Z skip: Need at least 2 CUDA devices (2.080s)
2022-11-23T01:38:25.8487914Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T01:38:25.8489063Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28434
2022-11-23T01:38:25.8489788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28435
2022-11-23T01:38:25.8490350Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28436
2022-11-23T01:38:25.8490869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28437
2022-11-23T01:38:25.8491644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8492292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8492985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8493548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8494109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8494841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8495509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8496190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8496723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8497229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8497964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8498472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8499144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8499665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8500203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8500958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8501487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8502147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8502693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8503182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8503766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8504298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8504823Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8505336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8505709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8506065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8506414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8506763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8507118Z dist init r=0, world=4
2022-11-23T01:38:25.8507303Z dist init r=2, world=4
2022-11-23T01:38:25.8507481Z dist init r=1, world=4
2022-11-23T01:38:25.8507644Z dist init r=3, world=4
2022-11-23T01:38:25.8507853Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:38:25.8508170Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T01:38:25.8508690Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28490
2022-11-23T01:38:25.8509088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28491
2022-11-23T01:38:25.8509507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28492
2022-11-23T01:38:25.8509834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28493
2022-11-23T01:38:25.8510275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8510611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8511043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8511379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8511717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8512189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8512521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8512937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8513282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8513618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8514113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8514597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8514925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8515355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8515689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8516025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8516520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8517024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8517341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8517768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8518114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8518438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8518938Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8519456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8519844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8520212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8520562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8520900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8521164Z dist init r=1, world=4
2022-11-23T01:38:25.8521334Z dist init r=2, world=4
2022-11-23T01:38:25.8521514Z dist init r=0, world=4
2022-11-23T01:38:25.8521689Z dist init r=3, world=4
2022-11-23T01:38:25.8521883Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:38:25.8522262Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T01:38:25.8522823Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28546
2022-11-23T01:38:25.8523211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28547
2022-11-23T01:38:25.8523539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28548
2022-11-23T01:38:25.8523861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28549
2022-11-23T01:38:25.8524315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8524633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8525064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8525413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8525757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8526213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8526542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8527063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8527399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8527736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8528205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8528533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8528946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8529291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8529634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8530113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8530608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8530933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8531359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8531694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8532026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8532520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8533071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8533573Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8533958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8534304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8534651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8535165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8535424Z dist init r=0, world=4
2022-11-23T01:38:25.8535606Z dist init r=2, world=4
2022-11-23T01:38:25.8535771Z dist init r=3, world=4
2022-11-23T01:38:25.8535945Z dist init r=1, world=4
2022-11-23T01:38:25.8536152Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:38:25.8536459Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T01:38:25.8537002Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28602
2022-11-23T01:38:25.8537395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28603
2022-11-23T01:38:25.8537728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28604
2022-11-23T01:38:25.8538120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28605
2022-11-23T01:38:25.8538580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8538915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8539336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8539685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8540026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8540499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8540815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8541245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8541591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8541914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8542383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8542712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8543139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8543471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8543807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8544303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8544806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8545121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8545552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8545962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8546284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8546785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8547306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8547822Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8548232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8548582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8548924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8549274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8549520Z dist init r=0, world=4
2022-11-23T01:38:25.8549706Z dist init r=1, world=4
2022-11-23T01:38:25.8549891Z dist init r=2, world=4
2022-11-23T01:38:25.8550055Z dist init r=3, world=4
2022-11-23T01:38:25.8550263Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:38:25.8550612Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T01:38:25.8551156Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28658
2022-11-23T01:38:25.8551557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28659
2022-11-23T01:38:25.8551893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28660
2022-11-23T01:38:25.8552225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28661
2022-11-23T01:38:25.8552665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8552998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8553428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8553774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8554099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8554569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8554895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8555308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8555654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8555993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8556462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8556777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8557206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8557555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8557877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8558349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8558708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8559146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8559476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8559812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8560306Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8560856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8561365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8561885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8562271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8562619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8562958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8563298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8563557Z dist init r=3, world=4
2022-11-23T01:38:25.8563728Z dist init r=1, world=4
2022-11-23T01:38:25.8563913Z dist init r=0, world=4
2022-11-23T01:38:25.8564094Z dist init r=2, world=4
2022-11-23T01:38:25.8564291Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:38:25.8564605Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-11-23T01:38:25.8565134Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28714
2022-11-23T01:38:25.8565529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28715
2022-11-23T01:38:25.8565846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28716
2022-11-23T01:38:25.8566166Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28717
2022-11-23T01:38:25.8566719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8567041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8567475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8567824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8568162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8568618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8568950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8569378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8569706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8570039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8570510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8570836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8571252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8571632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8571968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8572431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8572747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8573172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8573544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8573864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8574357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8575015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8575546Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8576049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8576438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8576790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8577139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8577471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8577726Z dist init r=1, world=4
2022-11-23T01:38:25.8577912Z dist init r=2, world=4
2022-11-23T01:38:25.8578080Z dist init r=3, world=4
2022-11-23T01:38:25.8578257Z dist init r=0, world=4
2022-11-23T01:38:25.8578465Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:38:25.8578801Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-11-23T01:38:25.8579356Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28770
2022-11-23T01:38:25.8579748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28771
2022-11-23T01:38:25.8580080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28772
2022-11-23T01:38:25.8580393Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28773
2022-11-23T01:38:25.8580838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8581169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8581603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8581939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8582273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:25.8582741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8583055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8583481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8583827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8584161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:25.8584668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8584998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8585425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8585753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8586083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:25.8586593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:25.8586918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:25.8587333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:25.8587678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:25.8588009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:25.8588503Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8589009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8589528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8590048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:25.8590433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:25.8590775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:25.8591115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:25.8591461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:25.8591710Z dist init r=3, world=4
2022-11-23T01:38:25.8591894Z dist init r=0, world=4
2022-11-23T01:38:25.8592073Z dist init r=2, world=4
2022-11-23T01:38:25.8592238Z dist init r=1, world=4
2022-11-23T01:38:25.8592444Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:38:25.8592582Z 
2022-11-23T01:38:25.8592783Z ----------------------------------------------------------------------
2022-11-23T01:38:25.8593013Z Ran 8 tests in 14.526s
2022-11-23T01:38:25.8593124Z 
2022-11-23T01:38:25.8593196Z OK (skipped=8)
2022-11-23T01:38:25.8593302Z 
2022-11-23T01:38:25.8593387Z Generating XML reports...
2022-11-23T01:38:25.8593841Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123013810.xml
2022-11-23T01:38:25.8594098Z 
2022-11-23T01:38:25.8594419Z ##[endgroup]
2022-11-23T01:38:25.8594869Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_onzwi1iw)
2022-11-23T01:38:25.8595127Z 
2022-11-23T01:38:25.8595330Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-11-23 01:38:25.845729]
2022-11-23T01:38:25.8595860Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:25.846088]
2022-11-23T01:38:42.0446207Z 
2022-11-23T01:38:42.0447214Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:38:42.0456202Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_yjz50imy)
2022-11-23T01:38:42.0457037Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-11-23T01:38:42.0457296Z 
2022-11-23T01:38:42.0457411Z Running tests...
2022-11-23T01:38:42.0457855Z ----------------------------------------------------------------------
2022-11-23T01:38:42.0458213Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0458779Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28837
2022-11-23T01:38:42.0459170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28838
2022-11-23T01:38:42.0459580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28839
2022-11-23T01:38:42.0459903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28840
2022-11-23T01:38:42.0460364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0460684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0461122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0461469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0461833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0462306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0462641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0463065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0463393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0463726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0464193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0464518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0464932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0465271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0465600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0466083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0466576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0466904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0467323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0467649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0467977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0468472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0468992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0469494Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0469878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0470263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0470592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0470934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0471186Z dist init r=3, world=4
2022-11-23T01:38:42.0471361Z dist init r=0, world=4
2022-11-23T01:38:42.0471524Z dist init r=1, world=4
2022-11-23T01:38:42.0471698Z dist init r=2, world=4
2022-11-23T01:38:42.0471902Z skip: Need at least 2 CUDA devices (2.178s)
2022-11-23T01:38:42.0472240Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0472782Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28893
2022-11-23T01:38:42.0473182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28894
2022-11-23T01:38:42.0473497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28895
2022-11-23T01:38:42.0473820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28896
2022-11-23T01:38:42.0474268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0474595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0475009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0475354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0475690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0476160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0476472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0476894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0477234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0477552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0478014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0478343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0478763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0479088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0479429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0479893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0480217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0480624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0480964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0481297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0481780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0482299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0482848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0483362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0483730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0484076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0484416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0484790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0485030Z dist init r=2, world=4
2022-11-23T01:38:42.0485213Z dist init r=3, world=4
2022-11-23T01:38:42.0485388Z dist init r=0, world=4
2022-11-23T01:38:42.0485550Z dist init r=1, world=4
2022-11-23T01:38:42.0485754Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:38:42.0486101Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0486759Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28949
2022-11-23T01:38:42.0487173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28950
2022-11-23T01:38:42.0487502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28951
2022-11-23T01:38:42.0487834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28952
2022-11-23T01:38:42.0488276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0488608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0489044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0489383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0489723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0490196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0490521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0490929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0491276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0491609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0492072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0492384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0492807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0493151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0493469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0493938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0494268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0494694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0495203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0495609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0496112Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0496634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0497145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0497661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0498080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0498410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0498752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0499089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0499345Z dist init r=1, world=4
2022-11-23T01:38:42.0499512Z dist init r=0, world=4
2022-11-23T01:38:42.0499691Z dist init r=3, world=4
2022-11-23T01:38:42.0499868Z dist init r=2, world=4
2022-11-23T01:38:42.0500060Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:38:42.0500403Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0500973Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29005
2022-11-23T01:38:42.0501373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29006
2022-11-23T01:38:42.0501684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29007
2022-11-23T01:38:42.0502005Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29008
2022-11-23T01:38:42.0502461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0502776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0503213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0503563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0503904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0504365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0504695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0505129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0505461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0505798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0506265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0506590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0507004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0507352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0507685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0508181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0508723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0509218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0509543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0509963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0510308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0510672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0511174Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0511687Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0512072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0512424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0512772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0513107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0513366Z dist init r=1, world=4
2022-11-23T01:38:42.0513551Z dist init r=0, world=4
2022-11-23T01:38:42.0513719Z dist init r=3, world=4
2022-11-23T01:38:42.0513897Z dist init r=2, world=4
2022-11-23T01:38:42.0514103Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:38:42.0514440Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0515009Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29061
2022-11-23T01:38:42.0515409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29062
2022-11-23T01:38:42.0515731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29063
2022-11-23T01:38:42.0516043Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29064
2022-11-23T01:38:42.0516489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0516818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0517227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0517568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0517901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0518368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0518677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0519098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0519440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0519772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0520223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0520549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0521004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0521332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0521661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0522120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0522443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0522856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0523224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0523556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0524048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0524556Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0525074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0525579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0525945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0526295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0526635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0527080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0527323Z dist init r=2, world=4
2022-11-23T01:38:42.0527503Z dist init r=1, world=4
2022-11-23T01:38:42.0527682Z dist init r=0, world=4
2022-11-23T01:38:42.0527843Z dist init r=3, world=4
2022-11-23T01:38:42.0528048Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:38:42.0528398Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T01:38:42.0528969Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29117
2022-11-23T01:38:42.0529360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29118
2022-11-23T01:38:42.0529687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29119
2022-11-23T01:38:42.0530007Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29120
2022-11-23T01:38:42.0530447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0530775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0531200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0531543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0531859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0532320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0532645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0533054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0533395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0533763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0534227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0534539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0535107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0535457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0535788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0536349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0536845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0537175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0537590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0537932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0538263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0538754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0539269Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0539782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0540165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0540515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0540846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0541185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0541442Z dist init r=0, world=4
2022-11-23T01:38:42.0541610Z dist init r=1, world=4
2022-11-23T01:38:42.0541784Z dist init r=2, world=4
2022-11-23T01:38:42.0541958Z dist init r=3, world=4
2022-11-23T01:38:42.0542149Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:38:42.0542542Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29173
2022-11-23T01:38:42.0542967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29174
2022-11-23T01:38:42.0543301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29175
2022-11-23T01:38:42.0543610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29176
2022-11-23T01:38:42.0544057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0544383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0544794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0545135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0545469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0545935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0546252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0546720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0547062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0547398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0547853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0548181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0548638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0548968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0549300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0549765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0550092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0550505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0550841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0551169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0551657Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0552168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0552685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0553196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0553584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0553918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0554252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0554599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0554843Z dist init r=1, world=4
2022-11-23T01:38:42.0555021Z dist init r=3, world=4
2022-11-23T01:38:42.0555198Z dist init r=0, world=4
2022-11-23T01:38:42.0555364Z dist init r=2, world=4
2022-11-23T01:38:42.0555568Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:38:42.0555972Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29229
2022-11-23T01:38:42.0556395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29230
2022-11-23T01:38:42.0556709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29231
2022-11-23T01:38:42.0557028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29232
2022-11-23T01:38:42.0557480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0557797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0558225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0558572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0558907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:42.0559401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0559730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0560156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0560485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0560817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:42.0561317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0561643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0562057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0562407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0562742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:42.0563208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:42.0563517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:42.0563949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:42.0564293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:42.0564611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:42.0565102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0565626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0566143Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0566646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:42.0567116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:42.0567461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:42.0567809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:42.0568139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:42.0568398Z dist init r=0, world=4
2022-11-23T01:38:42.0568578Z dist init r=1, world=4
2022-11-23T01:38:42.0568744Z dist init r=2, world=4
2022-11-23T01:38:42.0568921Z dist init r=3, world=4
2022-11-23T01:38:42.0569129Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:38:42.0569267Z 
2022-11-23T01:38:42.0569455Z ----------------------------------------------------------------------
2022-11-23T01:38:42.0569695Z Ran 8 tests in 14.621s
2022-11-23T01:38:42.0569805Z 
2022-11-23T01:38:42.0569874Z OK (skipped=8)
2022-11-23T01:38:42.0569978Z 
2022-11-23T01:38:42.0570061Z Generating XML reports...
2022-11-23T01:38:42.0570502Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123013827.xml
2022-11-23T01:38:42.0570765Z 
2022-11-23T01:38:42.0571075Z ##[endgroup]
2022-11-23T01:38:42.0571536Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_yjz50imy)
2022-11-23T01:38:42.0571789Z 
2022-11-23T01:38:42.0572054Z Running distributed/_tensor/test_redistribute ... [2022-11-23 01:38:42.045639]
2022-11-23T01:38:42.0572587Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:42.046094]
2022-11-23T01:38:59.5560343Z 
2022-11-23T01:38:59.5561353Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute
2022-11-23T01:38:59.5562481Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_qjg9hv8g)
2022-11-23T01:38:59.5563318Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute
2022-11-23T01:38:59.5563891Z 
2022-11-23T01:38:59.5564275Z Running tests...
2022-11-23T01:38:59.5564819Z ----------------------------------------------------------------------
2022-11-23T01:38:59.5565265Z   test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29296
2022-11-23T01:38:59.5565881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29297
2022-11-23T01:38:59.5566367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29298
2022-11-23T01:38:59.5566784Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29299
2022-11-23T01:38:59.5567111Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 29300
2022-11-23T01:38:59.5567573Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 29301
2022-11-23T01:38:59.5567950Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 29302
2022-11-23T01:38:59.5568259Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 29303
2022-11-23T01:38:59.5568789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5569125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5569610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5569962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5570348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5570858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5571336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5571962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5572400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5572978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:38:59.5573649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5574050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5574761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5575455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5575979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:38:59.5576831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5577356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5578138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5578948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5579546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5580335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5580949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5581683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5582218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5582602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5583141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5583658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5584254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5584583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5584906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:38:59.5585361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5585673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5586101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5586447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5586871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5587425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5587753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5588179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5588509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5588826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:38:59.5598615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5599025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5599384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5599757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5600137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:38:59.5600500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:38:59.5600852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:38:59.5601218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:38:59.5601771Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5602307Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5602941Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5603806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5604529Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5605284Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5606048Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5607007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:38:59.5607642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T01:38:59.5608209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2022-11-23T01:38:59.5608770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:38:59.5609365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:38:59.5610009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:38:59.5610555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T01:38:59.5611092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:38:59.5611864Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5612441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2022-11-23T01:38:59.5613138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5613888Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5614647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5615538Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5616348Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5617235Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5618027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:38:59.5618639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:38:59.5619191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2022-11-23T01:38:59.5619751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:38:59.5620302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2022-11-23T01:38:59.5620866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:38:59.5621406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2022-11-23T01:38:59.5622001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:38:59.5622619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2022-11-23T01:38:59.5623532Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5624353Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5625155Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5625978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5626797Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5627470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:38:59.5628047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2022-11-23T01:38:59.5628865Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5629666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5630411Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:38:59.5630979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2022-11-23T01:38:59.5631502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2022-11-23T01:38:59.5632012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:38:59.5632577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2022-11-23T01:38:59.5633131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:38:59.5633692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:38:59.5634447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5635059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:38:59.5635883Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5636494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2022-11-23T01:38:59.5637407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5638353Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5639305Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5640140Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5640833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2022-11-23T01:38:59.5641712Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5642593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:38:59.5643235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2022-11-23T01:38:59.5643845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2022-11-23T01:38:59.5644545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:38:59.5645171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:38:59.5645746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:38:59.5646746Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5647682Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5648679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5649340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2022-11-23T01:38:59.5649972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2022-11-23T01:38:59.5650638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2022-11-23T01:38:59.5651531Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5652484Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5653450Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5654187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2022-11-23T01:38:59.5654842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2022-11-23T01:38:59.5655902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5656845Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:38:59.5657559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2022-11-23T01:38:59.5658166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2022-11-23T01:38:59.5658789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2022-11-23T01:38:59.5659683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5660641Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5661585Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5662295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2022-11-23T01:38:59.5662974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2022-11-23T01:38:59.5663619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2022-11-23T01:38:59.5664505Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5665433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5666374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5667057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2022-11-23T01:38:59.5668088Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5668805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2022-11-23T01:38:59.5669683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:38:59.5670351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2022-11-23T01:38:59.5671014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2022-11-23T01:38:59.5671796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2022-11-23T01:38:59.5672712Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5673620Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5674560Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5675534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5676463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5677401Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5678371Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5679308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:38:59.5680394Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5681116Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5681745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2022-11-23T01:38:59.5682782Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5683524Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5684132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2022-11-23T01:38:59.5685104Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5685822Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5686445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2022-11-23T01:38:59.5687559Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5688320Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5688955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2022-11-23T01:38:59.5689977Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5690733Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5691447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2022-11-23T01:38:59.5692499Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5693250Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5693842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2022-11-23T01:38:59.5695056Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5695938Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5696529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2022-11-23T01:38:59.5697581Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5698336Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5698960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2022-11-23T01:38:59.5699863Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5700811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5701731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5702684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5703609Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5704632Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5705562Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5706510Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:38:59.5707233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2022-11-23T01:38:59.5707910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2022-11-23T01:38:59.5708566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2022-11-23T01:38:59.5709229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2022-11-23T01:38:59.5709858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2022-11-23T01:38:59.5710512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2022-11-23T01:38:59.5711160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2022-11-23T01:38:59.5711802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2022-11-23T01:38:59.5712699Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5713644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5714689Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5715612Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5716321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2022-11-23T01:38:59.5717214Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5718156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5719163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5719880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2022-11-23T01:38:59.5720539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2022-11-23T01:38:59.5721422Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:38:59.5722118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2022-11-23T01:38:59.5722783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2022-11-23T01:38:59.5723431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2022-11-23T01:38:59.5724090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2022-11-23T01:38:59.5724984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5725703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2022-11-23T01:38:59.5726625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5727428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2022-11-23T01:38:59.5728291Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5729210Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5729881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2022-11-23T01:38:59.5730744Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5731683Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5732390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2022-11-23T01:38:59.5733296Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5733986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2022-11-23T01:38:59.5735047Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:38:59.5735759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2022-11-23T01:38:59.5736431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2022-11-23T01:38:59.5737066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2022-11-23T01:38:59.5737870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2022-11-23T01:38:59.5738794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5739756Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5740470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2022-11-23T01:38:59.5741158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2022-11-23T01:38:59.5742170Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5743093Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5743780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2022-11-23T01:38:59.5744661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5745612Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5746572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5747265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2022-11-23T01:38:59.5747933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2022-11-23T01:38:59.5748617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2022-11-23T01:38:59.5749486Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:38:59.5750195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2022-11-23T01:38:59.5750846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2022-11-23T01:38:59.5751744Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5752669Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5753574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5754258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2022-11-23T01:38:59.5755075Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5755762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2022-11-23T01:38:59.5756431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2022-11-23T01:38:59.5757096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2022-11-23T01:38:59.5757951Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5758878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5759790Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5760434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2022-11-23T01:38:59.5761394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:38:59.5762093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2022-11-23T01:38:59.5762758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2022-11-23T01:38:59.5763378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2022-11-23T01:38:59.5764232Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5765222Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5766151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5767181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5768043Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5768991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5769901Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5770807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:38:59.5771509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1
2022-11-23T01:38:59.5772164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 3
2022-11-23T01:38:59.5772819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 5
2022-11-23T01:38:59.5773456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 7
2022-11-23T01:38:59.5774088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0
2022-11-23T01:38:59.5774735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 4
2022-11-23T01:38:59.5775531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 2
2022-11-23T01:38:59.5776191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 6
2022-11-23T01:38:59.5777090Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5778047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5778985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5779924Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5780853Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5781786Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5782512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0
2022-11-23T01:38:59.5783406Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5784499Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2022-11-23T01:38:59.5785200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 2
2022-11-23T01:38:59.5785849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 6
2022-11-23T01:38:59.5786478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 4
2022-11-23T01:38:59.5787131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1
2022-11-23T01:38:59.5787803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 3
2022-11-23T01:38:59.5788515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 5
2022-11-23T01:38:59.5789144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 7
2022-11-23T01:38:59.5790051Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5791007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5791965Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5792906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5793840Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5794794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5795742Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5796461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 5
2022-11-23T01:38:59.5797131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 2
2022-11-23T01:38:59.5797822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 6
2022-11-23T01:38:59.5798668Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2022-11-23T01:38:59.5799677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 4
2022-11-23T01:38:59.5800343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 7
2022-11-23T01:38:59.5800984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 3
2022-11-23T01:38:59.5801610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0
2022-11-23T01:38:59.5802244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1
2022-11-23T01:38:59.5803149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5804084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5805022Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5805744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0
2022-11-23T01:38:59.5806403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 5
2022-11-23T01:38:59.5807494Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5808170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1
2022-11-23T01:38:59.5809053Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5809770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 6
2022-11-23T01:38:59.5810654Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5811428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 4
2022-11-23T01:38:59.5812341Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5813271Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2022-11-23T01:38:59.5813984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 7
2022-11-23T01:38:59.5814632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 3
2022-11-23T01:38:59.5815394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 2
2022-11-23T01:38:59.5816267Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5817204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5818136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5819064Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5820009Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5820700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1
2022-11-23T01:38:59.5821342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 2
2022-11-23T01:38:59.5821989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 6
2022-11-23T01:38:59.5822618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 3
2022-11-23T01:38:59.5823505Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5824464Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5825172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 7
2022-11-23T01:38:59.5826084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2022-11-23T01:38:59.5826806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0
2022-11-23T01:38:59.5827489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 5
2022-11-23T01:38:59.5828131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 4
2022-11-23T01:38:59.5829009Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5829919Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5831010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5831956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5832853Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5833764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5834473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1
2022-11-23T01:38:59.5835242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 4
2022-11-23T01:38:59.5835913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 2
2022-11-23T01:38:59.5836474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 3
2022-11-23T01:38:59.5837054Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5837579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2022-11-23T01:38:59.5837965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0
2022-11-23T01:38:59.5838330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 5
2022-11-23T01:38:59.5838692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 7
2022-11-23T01:38:59.5839056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 6
2022-11-23T01:38:59.5839534Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5840058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5840950Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5841639Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5842148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5842666Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5843179Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5843699Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2022-11-23T01:38:59.5843972Z ok (4.786s)
2022-11-23T01:38:59.5844313Z   test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29552
2022-11-23T01:38:59.5844721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29553
2022-11-23T01:38:59.5845042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29554
2022-11-23T01:38:59.5845378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29555
2022-11-23T01:38:59.5845832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5846165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5846582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5847091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5847418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5847885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5848203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5848629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5848977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5849326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5849784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5850113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5850541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5850870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5851192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5851790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5852349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5853136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5853742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5854329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5855058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5855739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5856394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5857071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5857969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5858908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5859849Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5860779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5861279Z ok (1.821s)
2022-11-23T01:38:59.5861845Z   test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29608
2022-11-23T01:38:59.5862552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29609
2022-11-23T01:38:59.5863128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29610
2022-11-23T01:38:59.5863726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29611
2022-11-23T01:38:59.5864564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5865167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5865946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5866705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5867298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5868085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5868669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5869440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5870062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5870700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5871542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5872129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5872883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5873515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5874064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5874909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5875478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5876209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5876815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5877412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5878072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5878733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5879415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5880078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5881002Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5881991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5882965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5883924Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5884937Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5885695Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5886761Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5887520Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5888553Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5889325Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5890478Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:38:59.5891166Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:38:59.5891579Z ok (1.820s)
2022-11-23T01:38:59.5892131Z   test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29664
2022-11-23T01:38:59.5892813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29665
2022-11-23T01:38:59.5893430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29666
2022-11-23T01:38:59.5893974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29667
2022-11-23T01:38:59.5894787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5895675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5896468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5897096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5897669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5898462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5899022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5899777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5900390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5900962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5901784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5902376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5903126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5903715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5904289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5905162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5905726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5906495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5907122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5907679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5908322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5908976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5909626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5910251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5911169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5912122Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5913206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5914170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5914876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:38:59.5915535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:38:59.5916166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:38:59.5916940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:38:59.5917873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:38:59.5918851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:38:59.5919577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:38:59.5920196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:38:59.5921088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:38:59.5922014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:38:59.5922702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:38:59.5923338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:38:59.5924248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:38:59.5925192Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:38:59.5926112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:38:59.5927140Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:38:59.5927864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:38:59.5928528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:38:59.5929174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:38:59.5929852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:38:59.5930788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:38:59.5931712Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:38:59.5932434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:38:59.5933331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:38:59.5934279Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:38:59.5935143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:38:59.5935837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:38:59.5936605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:38:59.5937483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:38:59.5938397Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:38:59.5939307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:38:59.5940239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:38:59.5940863Z ok (1.921s)
2022-11-23T01:38:59.5941441Z   test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29744
2022-11-23T01:38:59.5942175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29745
2022-11-23T01:38:59.5942766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29746
2022-11-23T01:38:59.5943322Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29747
2022-11-23T01:38:59.5944147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5944735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5945494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5946111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5946692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5947551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5948177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5948934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5949553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5950151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5951009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5951634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5952445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5953091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5953649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5954508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5955107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5955883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5956487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5957042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5957701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5958348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5958989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5959944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5960631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5961517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5962415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5963365Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5963929Z ok (1.820s)
2022-11-23T01:38:59.5964506Z   test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29800
2022-11-23T01:38:59.5965203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29801
2022-11-23T01:38:59.5965782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29802
2022-11-23T01:38:59.5966346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29803
2022-11-23T01:38:59.5967258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5967810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5968563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5969153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5969735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.5970582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5971148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5971922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5972528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5973104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5973940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5974497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5975431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5976009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5976573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5977417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5978003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5978749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5979375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5979946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.5980602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.5981225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.5981880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.5982894Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5983562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.5984392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5985285Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5986276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.5986758Z ok (1.822s)
2022-11-23T01:38:59.5987352Z   test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29856
2022-11-23T01:38:59.5988070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29857
2022-11-23T01:38:59.5988664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29858
2022-11-23T01:38:59.5989238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29859
2022-11-23T01:38:59.5990067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5990651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5991428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5992069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5992621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:38:59.5993436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5994007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5994812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5995454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5996033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:38:59.5996852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.5997421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.5998196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.5998784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.5999371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:38:59.6000233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:38:59.6000811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:38:59.6001579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:38:59.6002187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:38:59.6002757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:38:59.6003378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:38:59.6004018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:38:59.6004674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:38:59.6005425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:38:59.6006355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.6007428Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.6008393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.6009415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:38:59.6009933Z ok (1.921s)
2022-11-23T01:38:59.6010117Z 
2022-11-23T01:38:59.6010476Z ----------------------------------------------------------------------
2022-11-23T01:38:59.6010904Z Ran 7 tests in 15.912s
2022-11-23T01:38:59.6011108Z 
2022-11-23T01:38:59.6011194Z OK
2022-11-23T01:38:59.6011362Z 
2022-11-23T01:38:59.6011514Z Generating XML reports...
2022-11-23T01:38:59.6012406Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123013843.xml
2022-11-23T01:38:59.6013516Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123013843.xml
2022-11-23T01:38:59.6013991Z 
2022-11-23T01:38:59.6014487Z ##[endgroup]
2022-11-23T01:38:59.6015545Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_qjg9hv8g)
2022-11-23T01:38:59.6016045Z 
2022-11-23T01:38:59.6016394Z Running distributed/test_pg_wrapper ... [2022-11-23 01:38:59.557040]
2022-11-23T01:38:59.6017345Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:38:59.557392]
2022-11-23T01:39:41.3279891Z 
2022-11-23T01:39:41.3280605Z Expand the folded group to see the log file of distributed/test_pg_wrapper
2022-11-23T01:39:41.3281571Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_fu3wupfs)
2022-11-23T01:39:41.3282132Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:39:41.3282491Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:39:41.3287454Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-11-23T01:39:41.3289406Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3289797Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3290135Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3290523Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3290889Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3291226Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3291535Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3292050Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3292402Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-11-23T01:39:41.3293341Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-11-23T01:39:41.3294203Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T01:39:41.3294509Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T01:39:41.3294849Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T01:39:41.3295397Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T01:39:41.3295887Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-11-23T01:39:41.3296480Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3297337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3297777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3298200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3298557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3298723Z 
2022-11-23T01:39:41.3298797Z Running tests...
2022-11-23T01:39:41.3299103Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3299499Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29933
2022-11-23T01:39:41.3299910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29934
2022-11-23T01:39:41.3300241Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29935
2022-11-23T01:39:41.3300554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29936
2022-11-23T01:39:41.3301007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3301345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3301774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3302108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3302431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3320244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3320884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3321536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3322012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3322610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3323444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3324023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3324740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3327628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3328253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3329154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3329786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3330293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3330637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3331047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3331414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:39:41.3331770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:39:41.3332139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:39:41.3332640Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3333027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:39:41.3333515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3334028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3334552Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3335145Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:39:41.3335660Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:39:41.3336474Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:39:41.3337003Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-11-23T01:39:41.3337513Z ok (2.275s)
2022-11-23T01:39:41.3338965Z 
2022-11-23T01:39:41.3339268Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3339693Z Ran 1 test in 2.276s
2022-11-23T01:39:41.3339906Z 
2022-11-23T01:39:41.3340024Z OK
2022-11-23T01:39:41.3340178Z 
2022-11-23T01:39:41.3340333Z Generating XML reports...
2022-11-23T01:39:41.3341131Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013902.xml
2022-11-23T01:39:41.3341648Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3342102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3342519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3342934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3343281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3343450Z 
2022-11-23T01:39:41.3343525Z Running tests...
2022-11-23T01:39:41.3343827Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3344230Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30024
2022-11-23T01:39:41.3344745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30025
2022-11-23T01:39:41.3345080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30026
2022-11-23T01:39:41.3345403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30027
2022-11-23T01:39:41.3345844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3346180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3346609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3346995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3347321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3347780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3348109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3348524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3348870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3349193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3349638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3349967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3350393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3350738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3351047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3351505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3351832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3352257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3352586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3352913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3353277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:39:41.3353632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:39:41.3353992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:39:41.3354487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3354884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:39:41.3355361Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3355872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3356389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3356670Z ok (2.174s)
2022-11-23T01:39:41.3356758Z 
2022-11-23T01:39:41.3356954Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3357194Z Ran 1 test in 2.174s
2022-11-23T01:39:41.3357307Z 
2022-11-23T01:39:41.3357368Z OK
2022-11-23T01:39:41.3357510Z 
2022-11-23T01:39:41.3357583Z Generating XML reports...
2022-11-23T01:39:41.3358060Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013905.xml
2022-11-23T01:39:41.3358574Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3359037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3359354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3359813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3360164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3360328Z 
2022-11-23T01:39:41.3360389Z Running tests...
2022-11-23T01:39:41.3360691Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3361113Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30115
2022-11-23T01:39:41.3361541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30116
2022-11-23T01:39:41.3361856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30117
2022-11-23T01:39:41.3362174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30118
2022-11-23T01:39:41.3362626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3362943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3363373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3363717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3364042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3364483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3364811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3365235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3365579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3365891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3366347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3366763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3367185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3367532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3367853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3368310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3368620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3369046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3369391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3369697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3369984Z skip: Need at least 4 CUDA devices (1.985s)
2022-11-23T01:39:41.3370120Z 
2022-11-23T01:39:41.3370363Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3370603Z Ran 1 test in 1.985s
2022-11-23T01:39:41.3370700Z 
2022-11-23T01:39:41.3370770Z OK (skipped=1)
2022-11-23T01:39:41.3370876Z 
2022-11-23T01:39:41.3370960Z Generating XML reports...
2022-11-23T01:39:41.3371437Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013909.xml
2022-11-23T01:39:41.3371933Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3372393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3372757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3373181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3373514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3373676Z 
2022-11-23T01:39:41.3373749Z Running tests...
2022-11-23T01:39:41.3374051Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3374479Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30170
2022-11-23T01:39:41.3375065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30171
2022-11-23T01:39:41.3375548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30172
2022-11-23T01:39:41.3376061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30173
2022-11-23T01:39:41.3376838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3380404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3381246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3381895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3382407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3382871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3383202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3383629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3383970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3384293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3384753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3385067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3385491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3385834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3386151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3386595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3386923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3387346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3387676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3388095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3388390Z skip: Need at least 4 CUDA devices (1.978s)
2022-11-23T01:39:41.3388530Z 
2022-11-23T01:39:41.3388733Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3388962Z Ran 1 test in 1.978s
2022-11-23T01:39:41.3389073Z 
2022-11-23T01:39:41.3389147Z OK (skipped=1)
2022-11-23T01:39:41.3389256Z 
2022-11-23T01:39:41.3389343Z Generating XML reports...
2022-11-23T01:39:41.3389805Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013913.xml
2022-11-23T01:39:41.3390369Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3390835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3391163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3391583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3391927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3392089Z 
2022-11-23T01:39:41.3392162Z Running tests...
2022-11-23T01:39:41.3392452Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3392884Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30225
2022-11-23T01:39:41.3393316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30226
2022-11-23T01:39:41.3393647Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30227
2022-11-23T01:39:41.3393957Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30228
2022-11-23T01:39:41.3394408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3394737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3395163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3395495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3395817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3396272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3396588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3397016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3397355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3397676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3398118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3398441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3398864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3399206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3399512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3399972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3400295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3400738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3401078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3401397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3401758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:39:41.3402114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:39:41.3402481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:39:41.3402875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:39:41.3403353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3403878Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3404393Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3404904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3405280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:39:41.3405643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:39:41.3406008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:39:41.3406370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:39:41.3406946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3407459Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3407970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3408483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3408753Z ok (2.275s)
2022-11-23T01:39:41.3408856Z 
2022-11-23T01:39:41.3409059Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3409304Z Ran 1 test in 2.275s
2022-11-23T01:39:41.3409418Z 
2022-11-23T01:39:41.3409588Z OK
2022-11-23T01:39:41.3409682Z 
2022-11-23T01:39:41.3409766Z Generating XML reports...
2022-11-23T01:39:41.3410251Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013916.xml
2022-11-23T01:39:41.3410760Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3411216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3411550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3411977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3412321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3412486Z 
2022-11-23T01:39:41.3412546Z Running tests...
2022-11-23T01:39:41.3412847Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3413257Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30328
2022-11-23T01:39:41.3413720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30329
2022-11-23T01:39:41.3414038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30330
2022-11-23T01:39:41.3414358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30331
2022-11-23T01:39:41.3414810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3415337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3416155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3416655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3416978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3417424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3417745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3418168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3418495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3418814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3419272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3419600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3420010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3420354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3420675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3421129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3421441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3421871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3422211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3422520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3422883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:39:41.3423248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:39:41.3423610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:39:41.3424092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3424607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3425003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:39:41.3425490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3425995Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3426282Z ok (2.175s)
2022-11-23T01:39:41.3426383Z 
2022-11-23T01:39:41.3426579Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3426802Z Ran 1 test in 2.175s
2022-11-23T01:39:41.3426912Z 
2022-11-23T01:39:41.3427019Z OK
2022-11-23T01:39:41.3427111Z 
2022-11-23T01:39:41.3427195Z Generating XML reports...
2022-11-23T01:39:41.3427670Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013920.xml
2022-11-23T01:39:41.3428168Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3428633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3428961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3429415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3429760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3429921Z 
2022-11-23T01:39:41.3429992Z Running tests...
2022-11-23T01:39:41.3430291Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3430698Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30419
2022-11-23T01:39:41.3431120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30420
2022-11-23T01:39:41.3431562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30421
2022-11-23T01:39:41.3431889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30422
2022-11-23T01:39:41.3432330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3432665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3433096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3433425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3433751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3434205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3434532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3434945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3435293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3435613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3436056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3436377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3436806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3437147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3437455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3437913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3438234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3438642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3438984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3439309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3439598Z skip: Need at least 4 CUDA devices (1.979s)
2022-11-23T01:39:41.3439729Z 
2022-11-23T01:39:41.3439952Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3440191Z Ran 1 test in 1.979s
2022-11-23T01:39:41.3440300Z 
2022-11-23T01:39:41.3440371Z OK (skipped=1)
2022-11-23T01:39:41.3440479Z 
2022-11-23T01:39:41.3440549Z Generating XML reports...
2022-11-23T01:39:41.3441022Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013924.xml
2022-11-23T01:39:41.3441535Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3442031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3442346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3442775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3443119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3443282Z 
2022-11-23T01:39:41.3443354Z Running tests...
2022-11-23T01:39:41.3443639Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3444070Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30474
2022-11-23T01:39:41.3444501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30475
2022-11-23T01:39:41.3444819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30476
2022-11-23T01:39:41.3445149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30477
2022-11-23T01:39:41.3445600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3445926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3446347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3446754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3447081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3447526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3447848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3448272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3448619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3448929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3449386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3449709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3450137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3450466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3450789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3451249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3451563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3451986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3452330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3452703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3452979Z skip: Need at least 4 CUDA devices (1.974s)
2022-11-23T01:39:41.3453114Z 
2022-11-23T01:39:41.3453309Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3453545Z Ran 1 test in 1.975s
2022-11-23T01:39:41.3453656Z 
2022-11-23T01:39:41.3453714Z OK (skipped=1)
2022-11-23T01:39:41.3453818Z 
2022-11-23T01:39:41.3453902Z Generating XML reports...
2022-11-23T01:39:41.3454375Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013927.xml
2022-11-23T01:39:41.3455061Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3455516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3455850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3456279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3456609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3456772Z 
2022-11-23T01:39:41.3456844Z Running tests...
2022-11-23T01:39:41.3457143Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3457565Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30529
2022-11-23T01:39:41.3457978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30530
2022-11-23T01:39:41.3458304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30531
2022-11-23T01:39:41.3458630Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30532
2022-11-23T01:39:41.3459084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3459398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3459823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3460165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3460473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:39:41.3460930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3461260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3461689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3462018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3462341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:39:41.3462803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3463115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3463547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3463889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3464213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:39:41.3464655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3464976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3465464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3465807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3466112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:39:41.3466471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:39:41.3466836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:39:41.3467184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:39:41.3467583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:39:41.3468076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3468600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3469100Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3469619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:39:41.3470013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:39:41.3470375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:39:41.3470723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:39:41.3471078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:39:41.3471564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3472079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3472582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3473095Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:39:41.3473382Z ok (2.279s)
2022-11-23T01:39:41.3473588Z 
2022-11-23T01:39:41.3473778Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3474019Z Ran 1 test in 2.280s
2022-11-23T01:39:41.3474131Z 
2022-11-23T01:39:41.3474191Z OK
2022-11-23T01:39:41.3474280Z 
2022-11-23T01:39:41.3474369Z Generating XML reports...
2022-11-23T01:39:41.3474836Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013931.xml
2022-11-23T01:39:41.3475386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3475716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3476132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3476476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3476643Z 
2022-11-23T01:39:41.3476718Z Running tests...
2022-11-23T01:39:41.3477019Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3477524Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3477906Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:39:41.3478056Z 
2022-11-23T01:39:41.3478288Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3478525Z Ran 1 test in 0.000s
2022-11-23T01:39:41.3478623Z 
2022-11-23T01:39:41.3478694Z OK (skipped=1)
2022-11-23T01:39:41.3478800Z 
2022-11-23T01:39:41.3478882Z Generating XML reports...
2022-11-23T01:39:41.3479349Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013935.xml
2022-11-23T01:39:41.3479888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3480246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3480671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3481012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3481160Z 
2022-11-23T01:39:41.3481230Z Running tests...
2022-11-23T01:39:41.3481527Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3482064Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3482440Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:39:41.3482593Z 
2022-11-23T01:39:41.3482786Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3483021Z Ran 1 test in 0.000s
2022-11-23T01:39:41.3483131Z 
2022-11-23T01:39:41.3483203Z OK (skipped=1)
2022-11-23T01:39:41.3483299Z 
2022-11-23T01:39:41.3483381Z Generating XML reports...
2022-11-23T01:39:41.3483851Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013936.xml
2022-11-23T01:39:41.3484396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3484711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3485137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3485492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3485663Z 
2022-11-23T01:39:41.3485733Z Running tests...
2022-11-23T01:39:41.3486018Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3486567Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3487041Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:39:41.3487191Z 
2022-11-23T01:39:41.3487385Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3487608Z Ran 1 test in 0.000s
2022-11-23T01:39:41.3487718Z 
2022-11-23T01:39:41.3487790Z OK (skipped=1)
2022-11-23T01:39:41.3487897Z 
2022-11-23T01:39:41.3487978Z Generating XML reports...
2022-11-23T01:39:41.3488428Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013938.xml
2022-11-23T01:39:41.3488972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3489299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3489724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3490057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3490220Z 
2022-11-23T01:39:41.3490294Z Running tests...
2022-11-23T01:39:41.3490588Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3491145Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3491538Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:39:41.3491690Z 
2022-11-23T01:39:41.3491883Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3492118Z Ran 1 test in 0.000s
2022-11-23T01:39:41.3492227Z 
2022-11-23T01:39:41.3492285Z OK (skipped=1)
2022-11-23T01:39:41.3492389Z 
2022-11-23T01:39:41.3492472Z Generating XML reports...
2022-11-23T01:39:41.3492938Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013939.xml
2022-11-23T01:39:41.3493518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:39:41.3493835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:39:41.3494261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:39:41.3494607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:39:41.3494769Z 
2022-11-23T01:39:41.3494827Z Running tests...
2022-11-23T01:39:41.3495252Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3495799Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-11-23T01:39:41.3496198Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:39:41.3496336Z 
2022-11-23T01:39:41.3496530Z ----------------------------------------------------------------------
2022-11-23T01:39:41.3496766Z Ran 1 test in 0.001s
2022-11-23T01:39:41.3496876Z 
2022-11-23T01:39:41.3496948Z OK (skipped=1)
2022-11-23T01:39:41.3497054Z 
2022-11-23T01:39:41.3497125Z Generating XML reports...
2022-11-23T01:39:41.3497594Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013940.xml
2022-11-23T01:39:41.3497870Z 
2022-11-23T01:39:41.3498337Z ##[endgroup]
2022-11-23T01:39:41.3498769Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_fu3wupfs)
2022-11-23T01:39:41.3499015Z 
2022-11-23T01:39:41.3499206Z Running distributed/_tensor/test_common_rules ... [2022-11-23 01:39:41.328826]
2022-11-23T01:39:41.3499725Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:39:41.329217]
2022-11-23T01:40:04.1032938Z 
2022-11-23T01:40:04.1033953Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules
2022-11-23T01:40:04.1034970Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_xtg3s78j)
2022-11-23T01:40:04.1040282Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules
2022-11-23T01:40:04.1040685Z 
2022-11-23T01:40:04.1040967Z Running tests...
2022-11-23T01:40:04.1041524Z ----------------------------------------------------------------------
2022-11-23T01:40:04.1042217Z   test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30682
2022-11-23T01:40:04.1042887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30683
2022-11-23T01:40:04.1043473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30684
2022-11-23T01:40:04.1043992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30685
2022-11-23T01:40:04.1044738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1045305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1046372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1047008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1047568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1048302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1048834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1049712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1050233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1050676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1051306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1051769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1052381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1052884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1053382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1054129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1054624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1055466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1056004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1056466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1056974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1057475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1058173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1058732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1059246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1059916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1060656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1061392Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1061780Z ok (2.070s)
2022-11-23T01:40:04.1062238Z   test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30738
2022-11-23T01:40:04.1062792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30739
2022-11-23T01:40:04.1063274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30740
2022-11-23T01:40:04.1063743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30741
2022-11-23T01:40:04.1064416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1064915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1065682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1066182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1066669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1067360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1067820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1068542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1069039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1069514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1070219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1070686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1071326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1071851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1072341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1073025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1073490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1074110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1074619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1075076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1075577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1076069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1076652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1077192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1077904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1078647Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1079439Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1080286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1080909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:40:04.1081499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:40:04.1082066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:40:04.1082506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:40:04.1082999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1083599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1083996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:40:04.1084495Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1084996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1085387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:40:04.1085793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:40:04.1086143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:40:04.1086701Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1087219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1087728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1088224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1088614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:40:04.1088978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:40:04.1089343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:40:04.1089697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:40:04.1090186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1090699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1091088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:40:04.1091438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:40:04.1091921Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1092441Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1092834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:40:04.1093182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:40:04.1093669Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1094178Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1094671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1095336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1095631Z ok (1.920s)
2022-11-23T01:40:04.1095948Z   test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30818
2022-11-23T01:40:04.1096320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30819
2022-11-23T01:40:04.1096711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30820
2022-11-23T01:40:04.1097042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30821
2022-11-23T01:40:04.1097497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1097815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1098244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1098590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1098943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1099407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1099737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1100174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1100504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1100825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1101285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1101609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1102036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1102383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1102707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1103153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1103479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1103906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1104250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1104556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1104921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1105290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1105645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1106330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1106919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1107411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1107917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1108432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1108825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:40:04.1109189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:40:04.1109540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:40:04.1109941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:40:04.1110423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1110942Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1111323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:40:04.1111689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:40:04.1112210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1112708Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1113105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:40:04.1113470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:40:04.1113957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1114457Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1114852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:40:04.1115341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1115850Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1116230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:40:04.1116593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:40:04.1116957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:40:04.1117446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1117943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1118336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:40:04.1118695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:40:04.1119185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1119679Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1120068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:40:04.1120430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:40:04.1120900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1121414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1121924Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1122465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1122740Z ok (2.021s)
2022-11-23T01:40:04.1123058Z   test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30898
2022-11-23T01:40:04.1123446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30899
2022-11-23T01:40:04.1123777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30900
2022-11-23T01:40:04.1124091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30901
2022-11-23T01:40:04.1124540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1124909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1125324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1125674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1125999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1126459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1126839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1127267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1127610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1127936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1128384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1128711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1129139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1129471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1129798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1130257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1130834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1131256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1131602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1131930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1132294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1132653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1133016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1133381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1133854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1134381Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1135032Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1135622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1135996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:40:04.1136359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:40:04.1136720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:40:04.1137082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:40:04.1137559Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1138124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1138516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:40:04.1139007Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1139512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1139902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:40:04.1140268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:40:04.1140616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:40:04.1141105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1141620Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1142136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1142632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1143021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:40:04.1143388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:40:04.1143748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:40:04.1144102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:40:04.1144586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1145094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1145486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:40:04.1145954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1146462Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1146845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:40:04.1147211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:40:04.1147555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:40:04.1148040Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1148575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1149085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1149576Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1149866Z ok (1.920s)
2022-11-23T01:40:04.1150194Z   test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30978
2022-11-23T01:40:04.1150606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30979
2022-11-23T01:40:04.1150941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30980
2022-11-23T01:40:04.1151268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30981
2022-11-23T01:40:04.1151722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1152042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1152473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1152822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1153136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1153601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1153936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1154364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1154701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1155025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1155489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1155822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1156234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1156579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1156905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1157347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1157672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1158097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1158449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1158759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1159125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1159490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1159846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1160208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1160696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1161248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1161759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1162273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1162561Z ok (1.820s)
2022-11-23T01:40:04.1162886Z   test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31034
2022-11-23T01:40:04.1163297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31035
2022-11-23T01:40:04.1163627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31036
2022-11-23T01:40:04.1163954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31037
2022-11-23T01:40:04.1164395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1164728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1165158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1165510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1165825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1166289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1166708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1167146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1167479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1167803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1168263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1168581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1169006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1169352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1169677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1170117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1170449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1170880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1171210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1171535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1171901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1172273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1172628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1173122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1173524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1174059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1174565Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1175241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1175535Z ok (1.821s)
2022-11-23T01:40:04.1175868Z   test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31090
2022-11-23T01:40:04.1176345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31091
2022-11-23T01:40:04.1176677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31092
2022-11-23T01:40:04.1177006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31093
2022-11-23T01:40:04.1177456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1177787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1178216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1178563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1178873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1179329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1179656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1180068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1180417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1180739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1181195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1181509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1181935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1182278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1182586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1183040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1183364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1183791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1184119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1184443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1184811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1185183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1185538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1185903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1186390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1186938Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1187465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1187985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1188380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:40:04.1188758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:40:04.1189119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:40:04.1189480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:40:04.1189969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1190472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1190865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:40:04.1191351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1191869Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1192246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:40:04.1192610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:40:04.1192976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:40:04.1193458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1193954Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1194348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:40:04.1194830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1195325Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1195716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:40:04.1196083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:40:04.1196445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:40:04.1196921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1197432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1197824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:40:04.1198189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:40:04.1198661Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1199203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1199596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:40:04.1199960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:40:04.1200431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1200941Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1201478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1201982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1202249Z ok (2.021s)
2022-11-23T01:40:04.1202581Z   test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31170
2022-11-23T01:40:04.1202984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31171
2022-11-23T01:40:04.1203301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31172
2022-11-23T01:40:04.1203628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31173
2022-11-23T01:40:04.1204077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1204407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1204821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1205166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1205486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1205933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1206260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1206760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1207108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1207414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1207880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1208207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1208636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1208968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1209291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1209749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1210061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1210486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1210833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1211154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1211504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1211914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1212276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1212769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1213150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1213642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1214193Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1214696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1215204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:40:04.1215571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:40:04.1215933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:40:04.1216280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:40:04.1216768Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1217279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1217671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:40:04.1218142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1218652Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-11-23T01:40:04.1219041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:40:04.1219406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:40:04.1219759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:40:04.1220252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1220771Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1221158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:40:04.1221632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1222139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2022-11-23T01:40:04.1222526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:40:04.1222873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:40:04.1223359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1223757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:40:04.1224238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1224612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:40:04.1225162Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1225672Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2022-11-23T01:40:04.1226061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:40:04.1226409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:40:04.1226767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:40:04.1227302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1227808Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1228307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1228819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2022-11-23T01:40:04.1229107Z ok (1.924s)
2022-11-23T01:40:04.1229420Z   test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31250
2022-11-23T01:40:04.1229813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31251
2022-11-23T01:40:04.1230147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31252
2022-11-23T01:40:04.1230473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31253
2022-11-23T01:40:04.1230909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1231246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1231674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1232024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1232331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1232790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1233119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1233538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1233882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1234206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1234666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1234981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1235406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1235750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1236059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1236519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1236845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1237276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1237632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1237955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1238317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1238685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1239032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1239392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1239919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1240425Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1240943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1241460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1241756Z ok (1.920s)
2022-11-23T01:40:04.1242064Z   test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31306
2022-11-23T01:40:04.1242456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31307
2022-11-23T01:40:04.1242790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31308
2022-11-23T01:40:04.1243118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31309
2022-11-23T01:40:04.1243561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1243895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1244322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1244652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1244975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1245430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1245757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1246171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1246572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1246902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1247351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1247680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1248107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1248451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1248759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1249220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1249548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1249975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1250339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1250664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1251026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1251379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1251744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1252134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1252629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1253138Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1253659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1254173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1254457Z ok (1.920s)
2022-11-23T01:40:04.1254752Z   test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31362
2022-11-23T01:40:04.1255312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31363
2022-11-23T01:40:04.1255646Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31364
2022-11-23T01:40:04.1255959Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31365
2022-11-23T01:40:04.1256414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1256749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1257182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1257516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1257839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:04.1258297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1258610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1259039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1259382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1259708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:04.1260154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1260484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1260910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1261254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1261563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:04.1262024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:04.1262351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:04.1262766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:04.1263165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:04.1263490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:04.1263852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:04.1264208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:04.1264704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1265141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:04.1265507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:04.1265992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1266510Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1267020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:04.1267291Z ok (1.821s)
2022-11-23T01:40:04.1267397Z 
2022-11-23T01:40:04.1267599Z ----------------------------------------------------------------------
2022-11-23T01:40:04.1267840Z Ran 11 tests in 21.180s
2022-11-23T01:40:04.1267956Z 
2022-11-23T01:40:04.1268019Z OK
2022-11-23T01:40:04.1268100Z 
2022-11-23T01:40:04.1268188Z Generating XML reports...
2022-11-23T01:40:04.1268636Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123013942.xml
2022-11-23T01:40:04.1268889Z 
2022-11-23T01:40:04.1269249Z ##[endgroup]
2022-11-23T01:40:04.1269695Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_xtg3s78j)
2022-11-23T01:40:04.1269952Z 
2022-11-23T01:40:04.1270153Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-11-23 01:40:04.104389]
2022-11-23T01:40:04.1270670Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:40:04.104809]
2022-11-23T01:40:27.0973539Z 
2022-11-23T01:40:27.0974566Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:40:27.0977826Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc__te_7i0f)
2022-11-23T01:40:27.0980305Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-11-23T01:40:27.0980615Z 
2022-11-23T01:40:27.0980733Z Running tests...
2022-11-23T01:40:27.0981152Z ----------------------------------------------------------------------
2022-11-23T01:40:27.0981993Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.0982736Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31429
2022-11-23T01:40:27.1009585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31430
2022-11-23T01:40:27.1010133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31431
2022-11-23T01:40:27.1010650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31432
2022-11-23T01:40:27.1011594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1012127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1012993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1013336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1013674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1014154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1014480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1015030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1015450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1015786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1016256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1016568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1016993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1017336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1017657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1018124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1018453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1018878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1019205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1019539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1020033Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1020558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1021059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1021577Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1021965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1022298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1022638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1022973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1023229Z dist init r=2, world=4
2022-11-23T01:40:27.1023396Z dist init r=3, world=4
2022-11-23T01:40:27.1023573Z dist init r=1, world=4
2022-11-23T01:40:27.1023744Z dist init r=0, world=4
2022-11-23T01:40:27.1023935Z skip: Need at least 2 CUDA devices (2.079s)
2022-11-23T01:40:27.1024369Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1024924Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31485
2022-11-23T01:40:27.1025286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31486
2022-11-23T01:40:27.1025657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31487
2022-11-23T01:40:27.1025980Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31488
2022-11-23T01:40:27.1026436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1026768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1027182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1027562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1027894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1028364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1028678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1029106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1029447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1029765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1030231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1030560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1030986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1031313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1031643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1032134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1032615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1032938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1033364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1033705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1034025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1034520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1035038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1035550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1035918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1036262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1036602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1036929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1037185Z dist init r=0, world=4
2022-11-23T01:40:27.1037367Z dist init r=1, world=4
2022-11-23T01:40:27.1037601Z dist init r=2, world=4
2022-11-23T01:40:27.1037762Z dist init r=3, world=4
2022-11-23T01:40:27.1037967Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:40:27.1038440Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T01:40:27.1038939Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31541
2022-11-23T01:40:27.1039299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31542
2022-11-23T01:40:27.1039628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31543
2022-11-23T01:40:27.1039949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31544
2022-11-23T01:40:27.1040425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1040751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1041180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1041514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1041847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1042313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1042637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1043048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1043390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1043718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1044177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1044490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1044911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1045249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1045564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1046052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1046634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1046961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1047376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1047721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1048052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1048546Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1049054Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1049568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1049954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1050288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1050627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1050996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1051256Z dist init r=3, world=4
2022-11-23T01:40:27.1051422Z dist init r=2, world=4
2022-11-23T01:40:27.1051594Z dist init r=0, world=4
2022-11-23T01:40:27.1051767Z dist init r=1, world=4
2022-11-23T01:40:27.1051973Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1052402Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1052936Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31597
2022-11-23T01:40:27.1053298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31598
2022-11-23T01:40:27.1053610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31599
2022-11-23T01:40:27.1053929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31600
2022-11-23T01:40:27.1054390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1054702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1055241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1055584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1055919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1056373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1056699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1057131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1057474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1057794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1058259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1058585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1058996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1059340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1059671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1060138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1060446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1060865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1061204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1061522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1062013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1062533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1063044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1063592Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1063974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1064322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1064668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1065000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1065294Z dist init r=0, world=4
2022-11-23T01:40:27.1065469Z dist init r=2, world=4
2022-11-23T01:40:27.1065631Z dist init r=1, world=4
2022-11-23T01:40:27.1065804Z dist init r=3, world=4
2022-11-23T01:40:27.1066006Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1066427Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1066926Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31653
2022-11-23T01:40:27.1067291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31654
2022-11-23T01:40:27.1067618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31655
2022-11-23T01:40:27.1067926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31656
2022-11-23T01:40:27.1068376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1068705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1069127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1069461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1069792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1070257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1070567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1070991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1071332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1071665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1072117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1072442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1072867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1073205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1073522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1074019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1074509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1074821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1075245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1075584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1075945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1076422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1076939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1077452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1077860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1078192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1078527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1078868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1079110Z dist init r=2, world=4
2022-11-23T01:40:27.1079287Z dist init r=1, world=4
2022-11-23T01:40:27.1079460Z dist init r=0, world=4
2022-11-23T01:40:27.1079620Z dist init r=3, world=4
2022-11-23T01:40:27.1079825Z skip: Need at least 2 CUDA devices (1.723s)
2022-11-23T01:40:27.1080259Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T01:40:27.1080764Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31709
2022-11-23T01:40:27.1081113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31710
2022-11-23T01:40:27.1081441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31711
2022-11-23T01:40:27.1081763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31712
2022-11-23T01:40:27.1082214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1082529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1082955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1083297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1083620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1084086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1084408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1084833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1085160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1085488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1085951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1086270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1086770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1087116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1087450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1087924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1088454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1088779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1089204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1089532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1089860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1090401Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1090915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1091419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1091802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1092148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1092477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1092820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1093072Z dist init r=2, world=4
2022-11-23T01:40:27.1093249Z dist init r=3, world=4
2022-11-23T01:40:27.1093412Z dist init r=0, world=4
2022-11-23T01:40:27.1093701Z dist init r=1, world=4
2022-11-23T01:40:27.1093906Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1094340Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1094840Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31765
2022-11-23T01:40:27.1095341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31766
2022-11-23T01:40:27.1095666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31767
2022-11-23T01:40:27.1095993Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31768
2022-11-23T01:40:27.1096449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1096768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1097198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1097542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1097866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1098338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1098669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1099093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1099421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1099755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1100220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1100527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1101006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1101350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1101680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1102135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1102462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1102889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1103267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1103583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1104077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1104598Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1105100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1105610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1105985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1106334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1106681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1107019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1107266Z dist init r=1, world=4
2022-11-23T01:40:27.1107447Z dist init r=3, world=4
2022-11-23T01:40:27.1107622Z dist init r=0, world=4
2022-11-23T01:40:27.1107787Z dist init r=2, world=4
2022-11-23T01:40:27.1107991Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1108423Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1108922Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31821
2022-11-23T01:40:27.1109274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31822
2022-11-23T01:40:27.1109606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31823
2022-11-23T01:40:27.1109929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31824
2022-11-23T01:40:27.1110373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1110704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1111133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1111477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1111800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1112267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1112594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1113009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1113376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1113710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1114175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1114482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1114907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1115249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1115609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1116086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1116581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1116905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1117314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1117655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1117990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1118483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1118992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1119512Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1119901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1120246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1120574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1120910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1121166Z dist init r=0, world=4
2022-11-23T01:40:27.1121331Z dist init r=3, world=4
2022-11-23T01:40:27.1121507Z dist init r=1, world=4
2022-11-23T01:40:27.1121684Z dist init r=2, world=4
2022-11-23T01:40:27.1121873Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:40:27.1122306Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T01:40:27.1122811Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31877
2022-11-23T01:40:27.1123172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31878
2022-11-23T01:40:27.1123485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31879
2022-11-23T01:40:27.1123805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31880
2022-11-23T01:40:27.1124257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1124587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1125001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1125343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1125703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1126161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1126568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1127005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1127343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1127662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1128169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1128496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1128907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1129247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1129579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1130040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1130351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1130773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1131118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1131448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1131926Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1132445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1132961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1133472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1133840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1134189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1134535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1134949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1135203Z dist init r=0, world=4
2022-11-23T01:40:27.1135380Z dist init r=3, world=4
2022-11-23T01:40:27.1135544Z dist init r=1, world=4
2022-11-23T01:40:27.1135720Z dist init r=2, world=4
2022-11-23T01:40:27.1136027Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:40:27.1136459Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1136946Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31933
2022-11-23T01:40:27.1137310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31934
2022-11-23T01:40:27.1137641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31935
2022-11-23T01:40:27.1137953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31936
2022-11-23T01:40:27.1138468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1138797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1139223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1139554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1139889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1140362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1140727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1141138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1141476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1141807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1142259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1142581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1143004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1143343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1143660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1144126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1144449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1144873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1145198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1145528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1146018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1146525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1147040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1147553Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1147934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1148270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1148607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1148945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1149201Z dist init r=2, world=4
2022-11-23T01:40:27.1149368Z dist init r=3, world=4
2022-11-23T01:40:27.1149544Z dist init r=1, world=4
2022-11-23T01:40:27.1149718Z dist init r=0, world=4
2022-11-23T01:40:27.1149908Z skip: Need at least 2 CUDA devices (1.820s)
2022-11-23T01:40:27.1150339Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T01:40:27.1150868Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31989
2022-11-23T01:40:27.1151216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31990
2022-11-23T01:40:27.1151542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31991
2022-11-23T01:40:27.1151860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31992
2022-11-23T01:40:27.1152313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1152630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1153090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1153436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1153769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1154225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1154554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1154979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1155306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1155640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1156109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1156433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1156847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1157192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1157525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1158010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1158502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1158828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1159248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1159580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1159914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1160409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1160927Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1161428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1161813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1162159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1162502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1162834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1163085Z dist init r=3, world=4
2022-11-23T01:40:27.1163265Z dist init r=2, world=4
2022-11-23T01:40:27.1163429Z dist init r=0, world=4
2022-11-23T01:40:27.1163637Z dist init r=1, world=4
2022-11-23T01:40:27.1163842Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1164261Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T01:40:27.1164759Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32045
2022-11-23T01:40:27.1165118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32046
2022-11-23T01:40:27.1165476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32047
2022-11-23T01:40:27.1165785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32048
2022-11-23T01:40:27.1166234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1166642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1167110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1167458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1167789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:27.1168255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1168572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1168999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1169342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1169671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:27.1170121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1170450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1170873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1171197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1171530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:40:27.1171998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:27.1172319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:27.1172731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:27.1173072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:27.1173404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:40:27.1173893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1174399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1175016Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1175537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:40:27.1175907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:40:27.1176315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:40:27.1176654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:27.1176993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:27.1177238Z dist init r=0, world=4
2022-11-23T01:40:27.1177415Z dist init r=3, world=4
2022-11-23T01:40:27.1177589Z dist init r=1, world=4
2022-11-23T01:40:27.1177751Z dist init r=2, world=4
2022-11-23T01:40:27.1177956Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:40:27.1178092Z 
2022-11-23T01:40:27.1178375Z ----------------------------------------------------------------------
2022-11-23T01:40:27.1178602Z Ran 12 tests in 21.405s
2022-11-23T01:40:27.1178715Z 
2022-11-23T01:40:27.1178787Z OK (skipped=12)
2022-11-23T01:40:27.1178894Z 
2022-11-23T01:40:27.1178977Z Generating XML reports...
2022-11-23T01:40:27.1179413Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123014005.xml
2022-11-23T01:40:27.1179661Z 
2022-11-23T01:40:27.1180069Z ##[endgroup]
2022-11-23T01:40:27.1180528Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc__te_7i0f)
2022-11-23T01:40:27.1180787Z 
2022-11-23T01:40:27.1180981Z Running distributed/fsdp/test_fsdp_misc ... [2022-11-23 01:40:27.098142]
2022-11-23T01:40:27.1181480Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:40:27.098496]
2022-11-23T01:40:54.7664738Z 
2022-11-23T01:40:54.7665590Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc
2022-11-23T01:40:54.7666990Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_2c7xq2_q)
2022-11-23T01:40:54.7668129Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2022-11-23T01:40:54.7668699Z 
2022-11-23T01:40:54.7669328Z Running tests...
2022-11-23T01:40:54.7669790Z ----------------------------------------------------------------------
2022-11-23T01:40:54.7670106Z   test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7670496Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32112
2022-11-23T01:40:54.7670889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32113
2022-11-23T01:40:54.7730699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7731224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7731918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7732665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7733016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7733524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7733855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7734281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7734622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7735115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7735621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7736311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7736693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7737044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7737308Z dist init r=0, world=2
2022-11-23T01:40:54.7737492Z dist init r=1, world=2
2022-11-23T01:40:54.7737688Z skip: Need at least 2 CUDA devices (1.871s)
2022-11-23T01:40:54.7737935Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7738362Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32140
2022-11-23T01:40:54.7738740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32141
2022-11-23T01:40:54.7739196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7739529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7739946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7740293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7740631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7741102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7741417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7741843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7742186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7742521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7743000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7743517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7743903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7744235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7744495Z dist init r=1, world=2
2022-11-23T01:40:54.7744675Z dist init r=0, world=2
2022-11-23T01:40:54.7744881Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7745118Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7745495Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32168
2022-11-23T01:40:54.7745894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32169
2022-11-23T01:40:54.7746333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7746661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7747087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7747433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7747754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7748221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7748550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7748995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7749340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7749672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7750163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7750671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7751086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7751433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7751689Z dist init r=1, world=2
2022-11-23T01:40:54.7751856Z dist init r=0, world=2
2022-11-23T01:40:54.7752066Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7752318Z   test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7752666Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32196
2022-11-23T01:40:54.7753048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32197
2022-11-23T01:40:54.7753502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7753831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7754247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7754590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7754919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7755377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7755704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7756128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7756468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7756783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7757274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7757786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7758172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7758504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7758756Z dist init r=1, world=2
2022-11-23T01:40:54.7758936Z dist init r=0, world=2
2022-11-23T01:40:54.7759129Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7759382Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7759730Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32224
2022-11-23T01:40:54.7760083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32225
2022-11-23T01:40:54.7760534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7760862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7761321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7761656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7761989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7762455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7762766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7763191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7763567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7763903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7764380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7764902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7765289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7765636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7765879Z dist init r=0, world=2
2022-11-23T01:40:54.7766059Z dist init r=1, world=2
2022-11-23T01:40:54.7766268Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7766590Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7766942Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32252
2022-11-23T01:40:54.7767312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32253
2022-11-23T01:40:54.7767767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7768084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7768512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7768857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7769180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7769647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7769976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7770402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7770729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7771061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7771550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7772055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7772439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7772784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7773042Z dist init r=0, world=2
2022-11-23T01:40:54.7773208Z dist init r=1, world=2
2022-11-23T01:40:54.7773412Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7773846Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32280
2022-11-23T01:40:54.7774265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32281
2022-11-23T01:40:54.7774718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7775216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7775655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7775987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7776428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7776897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7777223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7777639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7777982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7778314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7778791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7779315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7779700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7780043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7780289Z dist init r=1, world=2
2022-11-23T01:40:54.7780469Z dist init r=0, world=2
2022-11-23T01:40:54.7780677Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7781089Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32308
2022-11-23T01:40:54.7781541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32309
2022-11-23T01:40:54.7781997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7782324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7782740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7783087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7783417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7783882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7784192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7784619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7784959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7785276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7785768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7786282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7786666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7787040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7787293Z dist init r=0, world=2
2022-11-23T01:40:54.7787472Z dist init r=1, world=2
2022-11-23T01:40:54.7787665Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7788061Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32336
2022-11-23T01:40:54.7788490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32337
2022-11-23T01:40:54.7788977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7789293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7789720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7790064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7790399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7790852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7791173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7791596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7791925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7792258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7792749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7793264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7793634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7793979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7794236Z dist init r=0, world=2
2022-11-23T01:40:54.7794403Z dist init r=1, world=2
2022-11-23T01:40:54.7794610Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7795030Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32364
2022-11-23T01:40:54.7795482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32365
2022-11-23T01:40:54.7795920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7796250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7796676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7797021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7797345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7806988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7807348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7807772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7808123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7808554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7809060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7809575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7809965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7810316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7810614Z dist init r=1, world=2
2022-11-23T01:40:54.7810781Z dist init r=0, world=2
2022-11-23T01:40:54.7810989Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7811328Z   test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32392
2022-11-23T01:40:54.7811696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32393
2022-11-23T01:40:54.7812151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7812476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7812890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7813236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7813572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7814040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7814350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7814776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7815277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7815617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7816100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7816618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7817008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7817342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7817606Z dist init r=1, world=2
2022-11-23T01:40:54.7817787Z dist init r=0, world=2
2022-11-23T01:40:54.7817997Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7818338Z   test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32420
2022-11-23T01:40:54.7818728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32421
2022-11-23T01:40:54.7819181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7819492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7819922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7820272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7820606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7821061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7821450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7821882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7822227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7822549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7823043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7823608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7823983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7824329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7824580Z dist init r=0, world=2
2022-11-23T01:40:54.7824761Z dist init r=1, world=2
2022-11-23T01:40:54.7824955Z skip: Need at least 2 CUDA devices (1.615s)
2022-11-23T01:40:54.7825212Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7825593Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32448
2022-11-23T01:40:54.7825977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32449
2022-11-23T01:40:54.7826428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7826760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7827185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7827514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7827854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7828326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7828636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7829063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7829405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7829742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7830220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7830736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7831125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7831469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7831714Z dist init r=1, world=2
2022-11-23T01:40:54.7831893Z dist init r=0, world=2
2022-11-23T01:40:54.7832097Z skip: Need at least 2 CUDA devices (1.614s)
2022-11-23T01:40:54.7832344Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7832721Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32476
2022-11-23T01:40:54.7833116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32477
2022-11-23T01:40:54.7833555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7833884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7834341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7834686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7835007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7835475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7835803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7836255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7836583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7836915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7837407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7837912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7838297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7838643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7838899Z dist init r=0, world=2
2022-11-23T01:40:54.7839068Z dist init r=1, world=2
2022-11-23T01:40:54.7839274Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7839525Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7839993Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32504
2022-11-23T01:40:54.7840396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32505
2022-11-23T01:40:54.7840846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7841173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7841587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7841932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7842266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7842722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7843048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7843473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7843815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7844132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7844629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7845146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7845528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7845859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7846112Z dist init r=0, world=2
2022-11-23T01:40:54.7846293Z dist init r=1, world=2
2022-11-23T01:40:54.7846553Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7846824Z   test_no_params (__main__.TestFSDPMisc)
2022-11-23T01:40:54.7847176Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32532
2022-11-23T01:40:54.7847554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32533
2022-11-23T01:40:54.7848010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7848337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7848762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7849124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7849466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:40:54.7849938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:40:54.7850265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:40:54.7850678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:40:54.7851022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:40:54.7851352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:40:54.7851831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7852348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:40:54.7852734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:40:54.7853078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:40:54.7853326Z dist init r=0, world=2
2022-11-23T01:40:54.7853503Z dist init r=1, world=2
2022-11-23T01:40:54.7853707Z skip: Need at least 2 CUDA devices (1.613s)
2022-11-23T01:40:54.7853843Z 
2022-11-23T01:40:54.7854069Z ----------------------------------------------------------------------
2022-11-23T01:40:54.7854299Z Ran 16 tests in 26.078s
2022-11-23T01:40:54.7854412Z 
2022-11-23T01:40:54.7854486Z OK (skipped=16)
2022-11-23T01:40:54.7854594Z 
2022-11-23T01:40:54.7854679Z Generating XML reports...
2022-11-23T01:40:54.7855261Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123014028.xml
2022-11-23T01:40:54.7855513Z 
2022-11-23T01:40:54.7855946Z ##[endgroup]
2022-11-23T01:40:54.7856398Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_2c7xq2_q)
2022-11-23T01:40:54.7856652Z 
2022-11-23T01:40:54.7856847Z Running distributed/_tensor/test_dtensor ... [2022-11-23 01:40:54.767107]
2022-11-23T01:40:54.7857343Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:40:54.767450]
2022-11-23T01:41:24.1909725Z 
2022-11-23T01:41:24.1910572Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor
2022-11-23T01:41:24.1912383Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor__r0l118m)
2022-11-23T01:41:24.1913439Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor
2022-11-23T01:41:24.1913756Z 
2022-11-23T01:41:24.1913868Z Running tests...
2022-11-23T01:41:24.1914858Z ----------------------------------------------------------------------
2022-11-23T01:41:24.1915688Z   test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32571
2022-11-23T01:41:24.1916636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32572
2022-11-23T01:41:24.1917207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32573
2022-11-23T01:41:24.1917756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32574
2022-11-23T01:41:24.1918291Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 32575
2022-11-23T01:41:24.1918823Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 32576
2022-11-23T01:41:24.1919495Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 32577
2022-11-23T01:41:24.1920024Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 32578
2022-11-23T01:41:24.1920830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1921399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1922125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1922707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1923325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:41:24.1923882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1924212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1924638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1924985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1925306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.1925762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1926079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1926585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1926931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1927238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:41:24.1927770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1928100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1928526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1928857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1929181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:41:24.1929642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1929956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1930378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1930745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1931096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.1931545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1931869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1932361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1932713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1933020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:41:24.1933481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1933807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1934272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1934611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1935245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.1935727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.1936040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.1936468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.1936814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.1937122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.1937490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.1937857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.1938218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.1938567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.1938929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:41:24.1939287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:41:24.1939645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:41:24.1940137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1940535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:41:24.1941030Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1941550Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1942054Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1942566Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1943088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1943589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1944101Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.1944493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:41:24.1944850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:41:24.1945274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:41:24.1945631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T01:41:24.1945987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2022-11-23T01:41:24.1946340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2022-11-23T01:41:24.1946683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T01:41:24.1947079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:41:24.1947633Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1948150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1948525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:41:24.1949012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1949520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1949905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:41:24.1950377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1950764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:41:24.1951248Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1951748Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1975799Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.1976228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2022-11-23T01:41:24.1976599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2022-11-23T01:41:24.1976954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2022-11-23T01:41:24.1977326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:41:24.1977683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2022-11-23T01:41:24.1978181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1978687Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1979082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:41:24.1979448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2022-11-23T01:41:24.1979937Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1980319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:41:24.1980802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1981446Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1981966Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1982460Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1982855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:41:24.1983217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2022-11-23T01:41:24.1983613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2022-11-23T01:41:24.1984101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.1984495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2022-11-23T01:41:24.1984853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:41:24.1985329Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1985838Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1986349Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1986860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1987235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2022-11-23T01:41:24.1987594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:41:24.1988078Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1988586Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1988963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:41:24.1989323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2022-11-23T01:41:24.1989801Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1990315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.1990694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:41:24.1991053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2022-11-23T01:41:24.1991408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:41:24.1991876Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1992318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2022-11-23T01:41:24.1992795Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1993182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2022-11-23T01:41:24.1993527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2022-11-23T01:41:24.1994039Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1994553Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1994939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2022-11-23T01:41:24.1995407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1995914Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1996452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1996953Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.1997329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2022-11-23T01:41:24.1997689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2022-11-23T01:41:24.1998044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2022-11-23T01:41:24.1998398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2022-11-23T01:41:24.1998738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2022-11-23T01:41:24.1999224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.1999733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2000235Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2000742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2001246Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2001632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2022-11-23T01:41:24.2001975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2022-11-23T01:41:24.2002458Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2002971Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2003357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2022-11-23T01:41:24.2003707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2022-11-23T01:41:24.2004188Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2004575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2022-11-23T01:41:24.2004932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2022-11-23T01:41:24.2005272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2022-11-23T01:41:24.2005628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2022-11-23T01:41:24.2006107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2006771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2007273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2007782Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2008173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2022-11-23T01:41:24.2008515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2022-11-23T01:41:24.2009030Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2009422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2022-11-23T01:41:24.2009778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2022-11-23T01:41:24.2010125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2022-11-23T01:41:24.2010608Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2011117Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2011502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2022-11-23T01:41:24.2011970Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2012358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2022-11-23T01:41:24.2012712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2022-11-23T01:41:24.2013195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2013689Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2014194Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2014697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2015289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2022-11-23T01:41:24.2015766Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2016155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2022-11-23T01:41:24.2016639Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2017135Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2017523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2022-11-23T01:41:24.2017882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2022-11-23T01:41:24.2018242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2022-11-23T01:41:24.2018714Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2022-11-23T01:41:24.2019103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2022-11-23T01:41:24.2019459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2022-11-23T01:41:24.2019880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2022-11-23T01:41:24.2020354Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2020744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2022-11-23T01:41:24.2021228Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2021615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2022-11-23T01:41:24.2022126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2022635Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2023148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2023655Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2024031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2022-11-23T01:41:24.2024510Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2025017Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2022-11-23T01:41:24.2025408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2022-11-23T01:41:24.2025762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2022-11-23T01:41:24.2026127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2022-11-23T01:41:24.2026489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2022-11-23T01:41:24.2026835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2022-11-23T01:41:24.2027320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2027835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2028342Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2028834Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2029334Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2029727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2022-11-23T01:41:24.2030087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2022-11-23T01:41:24.2030436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2022-11-23T01:41:24.2030800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2022-11-23T01:41:24.2031286Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2031791Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2032197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2022-11-23T01:41:24.2032682Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2022-11-23T01:41:24.2033068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2022-11-23T01:41:24.2033418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2022-11-23T01:41:24.2033779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2022-11-23T01:41:24.2034291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2034796Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2035174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2022-11-23T01:41:24.2035536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2022-11-23T01:41:24.2036024Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2036527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2037018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2037409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2022-11-23T01:41:24.2037894Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2038398Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2038887Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2022-11-23T01:41:24.2039270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2022-11-23T01:41:24.2039633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2022-11-23T01:41:24.2039994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2022-11-23T01:41:24.2040342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2022-11-23T01:41:24.2040702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2022-11-23T01:41:24.2041186Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2041695Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2042073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2022-11-23T01:41:24.2042436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2022-11-23T01:41:24.2042920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2043296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2022-11-23T01:41:24.2043781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2044291Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2044820Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2045196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2022-11-23T01:41:24.2045675Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2046183Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2022-11-23T01:41:24.2046669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2022-11-23T01:41:24.2047057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2022-11-23T01:41:24.2047417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2022-11-23T01:41:24.2047778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2022-11-23T01:41:24.2048263Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2048756Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2049253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2049759Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2050263Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2050754Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2051251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2051745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2022-11-23T01:41:24.2052026Z ok (3.285s)
2022-11-23T01:41:24.2052326Z   test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32755
2022-11-23T01:41:24.2052706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32756
2022-11-23T01:41:24.2053032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32757
2022-11-23T01:41:24.2053344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32758
2022-11-23T01:41:24.2053661Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 32759
2022-11-23T01:41:24.2053986Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 32760
2022-11-23T01:41:24.2054311Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 32761
2022-11-23T01:41:24.2054617Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 32762
2022-11-23T01:41:24.2055205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2055539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2055960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2056306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2056631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2057087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2057400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2057884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2058230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2058537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:41:24.2058993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2059322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2059799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2060130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2060447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2060906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2061230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2061642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2061985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2062304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:41:24.2062749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2063073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2063497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2063838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2064144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2064599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2064920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2065329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2066015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2066339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:41:24.2066790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2067101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2067526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2067866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2068184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:41:24.2068627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2068948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2069372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2069700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2070015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2070412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2070780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2071129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2071487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:41:24.2071840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2072191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:41:24.2072700Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2073097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:41:24.2073456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:41:24.2073932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2074442Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2074951Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2075465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2075964Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2076470Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2076980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2077373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T01:41:24.2077719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:41:24.2078074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2022-11-23T01:41:24.2078432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:41:24.2078785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:41:24.2079128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2022-11-23T01:41:24.2079481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T01:41:24.2079831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:41:24.2080313Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2080809Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2081321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2081830Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2082338Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2082834Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2083365Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2083755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:41:24.2084220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2084604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2022-11-23T01:41:24.2084959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:41:24.2085345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:41:24.2085688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2022-11-23T01:41:24.2086041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2022-11-23T01:41:24.2086473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2022-11-23T01:41:24.2086830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:41:24.2087305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2087696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:41:24.2088184Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2088697Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2089071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2022-11-23T01:41:24.2089556Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2090059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2090553Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2090934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:41:24.2091414Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2091915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2092288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2022-11-23T01:41:24.2092646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2022-11-23T01:41:24.2093008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:41:24.2093360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:41:24.2093700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2022-11-23T01:41:24.2094177Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2094688Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2095360Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2095919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2096428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2096814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2022-11-23T01:41:24.2097172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:41:24.2097519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:41:24.2097915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:41:24.2098393Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2098768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2022-11-23T01:41:24.2099251Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2099757Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2100150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2022-11-23T01:41:24.2100496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:41:24.2100853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2022-11-23T01:41:24.2101337Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2101846Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2102340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2102845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2103350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2103731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2022-11-23T01:41:24.2104199Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2104578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2022-11-23T01:41:24.2105059Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2105564Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2105936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2022-11-23T01:41:24.2106293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2022-11-23T01:41:24.2106650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2022-11-23T01:41:24.2106990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2022-11-23T01:41:24.2107345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2022-11-23T01:41:24.2107696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2022-11-23T01:41:24.2108202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2108702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2109209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2109716Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2110103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2022-11-23T01:41:24.2110470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2022-11-23T01:41:24.2110946Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2111454Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2111959Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2112450Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2112833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2022-11-23T01:41:24.2113195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2022-11-23T01:41:24.2113555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2022-11-23T01:41:24.2113898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2022-11-23T01:41:24.2114252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2022-11-23T01:41:24.2114606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2022-11-23T01:41:24.2115076Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2115584Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2116092Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2116599Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2117087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2117589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2118097Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2118601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2118869Z ok (3.064s)
2022-11-23T01:41:24.2119200Z   test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 447
2022-11-23T01:41:24.2119593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 448
2022-11-23T01:41:24.2119919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 449
2022-11-23T01:41:24.2120224Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 450
2022-11-23T01:41:24.2120537Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 451
2022-11-23T01:41:24.2120880Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 452
2022-11-23T01:41:24.2121181Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 453
2022-11-23T01:41:24.2121493Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 454
2022-11-23T01:41:24.2121945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2122275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2122691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2123062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2123384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2123831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2124157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2124584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2124924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2125230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:41:24.2125684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2126008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2126505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2126835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2127160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2127620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2127930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2128353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2128695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2129014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2129458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2129783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2130209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2130539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2130860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2131312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2131637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2132047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2132387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2132702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:41:24.2133153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2133499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2133925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2134267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2134575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:41:24.2135426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2135979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2136481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2136814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2137135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:41:24.2137502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2137855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2138218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:41:24.2138582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2138936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2139281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:41:24.2139636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:41:24.2139988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:41:24.2140473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2140975Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2141489Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2142002Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2142516Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2143011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2143517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2144025Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2144307Z ok (2.839s)
2022-11-23T01:41:24.2144623Z   test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 559
2022-11-23T01:41:24.2145021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 560
2022-11-23T01:41:24.2145341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 561
2022-11-23T01:41:24.2145651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 562
2022-11-23T01:41:24.2145963Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 563
2022-11-23T01:41:24.2146277Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 564
2022-11-23T01:41:24.2146634Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 565
2022-11-23T01:41:24.2146936Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 566
2022-11-23T01:41:24.2147382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2147708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2148123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2148522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2148845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2149301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2149612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2150038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2150382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2150689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:41:24.2151142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2151468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2151893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2152222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2152539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2152996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2153320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2153729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2154069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2154385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2154826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2155152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2155579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2155922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2156226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2156680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2157007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2157420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2157757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2158079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:41:24.2158535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2158845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2159306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2159647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2159965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:41:24.2160408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2160732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2161156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2161512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2161829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:41:24.2162192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2162559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2162905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2163266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2163619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:41:24.2163963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:41:24.2164320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:41:24.2164673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:41:24.2165160Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2165664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2166177Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2166771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2167285Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2167783Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2168292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2168801Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2169084Z ok (2.841s)
2022-11-23T01:41:24.2169393Z   test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 673
2022-11-23T01:41:24.2169780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 674
2022-11-23T01:41:24.2170104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 675
2022-11-23T01:41:24.2170409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 676
2022-11-23T01:41:24.2170727Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 677
2022-11-23T01:41:24.2171036Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 678
2022-11-23T01:41:24.2171347Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 679
2022-11-23T01:41:24.2171680Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 680
2022-11-23T01:41:24.2172122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2172447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2172874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2173208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2173533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2174032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2174345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2174770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2175687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2176035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T01:41:24.2176564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2176900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2177332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2177674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2177996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2178462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2178791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2179207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2179547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2179864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2180321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2180636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2181064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2181409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2181718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T01:41:24.2182170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2182493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2182915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2183243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2183562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T01:41:24.2184019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2184330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2184752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2185230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2185557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2186003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2186330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2186756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2187147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2187452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T01:41:24.2187811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2188179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2188531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2188891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2189251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T01:41:24.2189606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T01:41:24.2189951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2022-11-23T01:41:24.2190445Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2190843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2022-11-23T01:41:24.2191329Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2191829Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2192342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2192854Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2193351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2193859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2194369Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2022-11-23T01:41:24.2194766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T01:41:24.2195112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T01:41:24.2195470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:41:24.2195831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T01:41:24.2196182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2022-11-23T01:41:24.2196526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2022-11-23T01:41:24.2196878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:41:24.2197228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T01:41:24.2197740Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2198238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2198749Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2199261Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2199802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2200293Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2200807Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2201196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-11-23T01:41:24.2201554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:41:24.2201900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2022-11-23T01:41:24.2202255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2022-11-23T01:41:24.2202608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2022-11-23T01:41:24.2203083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2022-11-23T01:41:24.2203468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2022-11-23T01:41:24.2203826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2022-11-23T01:41:24.2204185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:41:24.2204654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2205039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:41:24.2205517Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2205905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2022-11-23T01:41:24.2206451Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2206843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2022-11-23T01:41:24.2207323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2207834Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2208329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2208841Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2209232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:41:24.2209700Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2022-11-23T01:41:24.2210120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2022-11-23T01:41:24.2210477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2022-11-23T01:41:24.2210833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2022-11-23T01:41:24.2211176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-11-23T01:41:24.2211658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2212166Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2212709Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2213205Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2213710Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2214100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2022-11-23T01:41:24.2214455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2022-11-23T01:41:24.2214800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2022-11-23T01:41:24.2215472Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2215986Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2216375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2022-11-23T01:41:24.2216855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2022-11-23T01:41:24.2217247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2022-11-23T01:41:24.2217605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2022-11-23T01:41:24.2217950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2022-11-23T01:41:24.2218309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2022-11-23T01:41:24.2218793Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2219306Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2219805Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2220192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2022-11-23T01:41:24.2220673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2221180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2221674Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2222182Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2222690Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2022-11-23T01:41:24.2223130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2022-11-23T01:41:24.2223479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2022-11-23T01:41:24.2223836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2022-11-23T01:41:24.2224196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2022-11-23T01:41:24.2224535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2022-11-23T01:41:24.2224927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2022-11-23T01:41:24.2225290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2022-11-23T01:41:24.2225767Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2226269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2226774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2227283Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2227788Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2228282Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2228788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2229296Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2022-11-23T01:41:24.2229681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2022-11-23T01:41:24.2230027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2022-11-23T01:41:24.2230385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2022-11-23T01:41:24.2230742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2022-11-23T01:41:24.2231103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2022-11-23T01:41:24.2231446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2022-11-23T01:41:24.2231800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2022-11-23T01:41:24.2232279Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2232656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2022-11-23T01:41:24.2233133Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2233635Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2234140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2234631Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2235132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2235672Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2236182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2022-11-23T01:41:24.2236452Z ok (3.038s)
2022-11-23T01:41:24.2236766Z   test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 833
2022-11-23T01:41:24.2237138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 834
2022-11-23T01:41:24.2237459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 835
2022-11-23T01:41:24.2237792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 836
2022-11-23T01:41:24.2238242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2238570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2238989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2239335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2239659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2240118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2240430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2240858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2241200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2241507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2241958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2242284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2242708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2243036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2243353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2243808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2244135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2244546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2244887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2245204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2245548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2245910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2246270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2246839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2247351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2247739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2248258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2248772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2249045Z ok (1.828s)
2022-11-23T01:41:24.2249352Z   test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 889
2022-11-23T01:41:24.2249726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 890
2022-11-23T01:41:24.2250032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 891
2022-11-23T01:41:24.2250377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 892
2022-11-23T01:41:24.2250825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2251150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2251565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2251909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2252225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2252673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2252998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2253426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2253769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2254079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2254535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2254864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2255415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2255743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2256063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2256522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2256837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2257259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2257602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2257921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2258266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2258631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2258994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2259338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2259831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2260349Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2260919Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2261419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2261705Z ok (1.820s)
2022-11-23T01:41:24.2262018Z   test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 945
2022-11-23T01:41:24.2262400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 946
2022-11-23T01:41:24.2262708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 947
2022-11-23T01:41:24.2263078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 948
2022-11-23T01:41:24.2263526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2263842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2264271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2264617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2264935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2265377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2265700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2266125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2266469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2266773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2267228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2267548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2267960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2268298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2268617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2269069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2269381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2269807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2270148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2270456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2270821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2271190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2271550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2271897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2272387Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2272906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2273446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2273947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2274233Z ok (1.720s)
2022-11-23T01:41:24.2274531Z   test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1001
2022-11-23T01:41:24.2274889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1002
2022-11-23T01:41:24.2275212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1003
2022-11-23T01:41:24.2275556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1004
2022-11-23T01:41:24.2276002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2276315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2276744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2277092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2277413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2277856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2278182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2278608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2278939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2279256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2279710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2280035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2280446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2280785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2281102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2281540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2281866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2282289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2282627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2282933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2283295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2283659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2284023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2284373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2284868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2285381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2285913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2286483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2286771Z ok (1.820s)
2022-11-23T01:41:24.2287063Z   test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1057
2022-11-23T01:41:24.2287411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1058
2022-11-23T01:41:24.2287735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1059
2022-11-23T01:41:24.2288091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1060
2022-11-23T01:41:24.2288542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2288856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2289281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2289627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2289937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2290391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2290716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2291142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2291471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2291791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2292244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2292558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2292984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2293326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2293644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2294082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2294407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2294828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2295336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2295646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2296010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2296373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2296721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2297082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2297573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2298092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2298593Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2299172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2299461Z ok (1.820s)
2022-11-23T01:41:24.2299770Z   test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1113
2022-11-23T01:41:24.2300131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1114
2022-11-23T01:41:24.2300458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1115
2022-11-23T01:41:24.2300819Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1116
2022-11-23T01:41:24.2301252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2301579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2302006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2302351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2302658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2303112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2303435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2303846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2304190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2304508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2304958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2305270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2305690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2306028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2306343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2306781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2307109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2307530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2307857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2308177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2308537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2308898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2309376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2309772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2310137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2310625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2311125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2311667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2311953Z ok (1.920s)
2022-11-23T01:41:24.2312228Z   test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1169
2022-11-23T01:41:24.2312586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1170
2022-11-23T01:41:24.2312909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1171
2022-11-23T01:41:24.2313230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1172
2022-11-23T01:41:24.2313691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2314019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2314450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2314781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2315100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:41:24.2315554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2315879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2316291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2316635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2316954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:41:24.2317405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2317719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2318141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2318482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2318787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:41:24.2319242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:41:24.2319566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:41:24.2319990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:41:24.2320315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:41:24.2320637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:41:24.2320995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:41:24.2321347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:41:24.2321706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:41:24.2322062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:41:24.2322542Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2323051Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2323566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2324104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:41:24.2324386Z ok (1.820s)
2022-11-23T01:41:24.2324477Z 
2022-11-23T01:41:24.2324678Z ----------------------------------------------------------------------
2022-11-23T01:41:24.2324923Z Ran 12 tests in 27.817s
2022-11-23T01:41:24.2325037Z 
2022-11-23T01:41:24.2325097Z OK
2022-11-23T01:41:24.2325187Z 
2022-11-23T01:41:24.2325258Z Generating XML reports...
2022-11-23T01:41:24.2325690Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123014056.xml
2022-11-23T01:41:24.2326262Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123014056.xml
2022-11-23T01:41:24.2326571Z 
2022-11-23T01:41:24.2326919Z ##[endgroup]
2022-11-23T01:41:24.2327365Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor__r0l118m)
2022-11-23T01:41:24.2327618Z 
2022-11-23T01:41:24.2327818Z Running distributed/_tensor/test_tensor_ops ... [2022-11-23 01:41:24.192614]
2022-11-23T01:41:24.2328334Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:41:24.192965]
2022-11-23T01:42:03.1745115Z 
2022-11-23T01:42:03.1745826Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2022-11-23T01:42:03.1748803Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_bz9tht5j)
2022-11-23T01:42:03.1749512Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2022-11-23T01:42:03.1749723Z 
2022-11-23T01:42:03.1749784Z Running tests...
2022-11-23T01:42:03.1750100Z ----------------------------------------------------------------------
2022-11-23T01:42:03.1750496Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1236
2022-11-23T01:42:03.1750882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1237
2022-11-23T01:42:03.1751198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1238
2022-11-23T01:42:03.1751513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1239
2022-11-23T01:42:03.1751977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1752317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1752740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1753090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1753413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1753863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1754194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1754622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1754963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1755274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1755737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1756062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1756660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1757007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1757329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1757791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1758106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1758530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1758934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1759256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1759607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1759977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1760343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1760826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1761219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1761705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1762216Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1762720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1763007Z ok (2.064s)
2022-11-23T01:42:03.1763309Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1292
2022-11-23T01:42:03.1763676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1293
2022-11-23T01:42:03.1763988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1294
2022-11-23T01:42:03.1764429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1295
2022-11-23T01:42:03.1764877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1765198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1765620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1765966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1766363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1766808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1767132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1767555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1767884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1768205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1768665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1768989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1769404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1769786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1770107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1770652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1770966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1771392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1771772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1772081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1772439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1772807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1773298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1773678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1774035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1774525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1775326Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1776006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1776484Z ok (1.820s)
2022-11-23T01:42:03.1776950Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1348
2022-11-23T01:42:03.1777559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1349
2022-11-23T01:42:03.1778103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1350
2022-11-23T01:42:03.1778652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1351
2022-11-23T01:42:03.1779486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1780053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1780865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1781255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1781577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1782030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1782361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1782786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1783118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1783440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1783905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1784266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1784928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1785376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1794786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1795324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1795661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1796082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1796548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1796874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1797227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1797605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1797975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1798338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1798824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1799342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1799867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1800377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1800650Z ok (1.920s)
2022-11-23T01:42:03.1800956Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1404
2022-11-23T01:42:03.1801329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1405
2022-11-23T01:42:03.1801643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1406
2022-11-23T01:42:03.1801969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1407
2022-11-23T01:42:03.1802598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1802931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1803349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1803695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1804019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1804478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1804793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1805221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1805564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1805869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1806405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1806739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1807165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1807542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1807869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1808323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1808647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1809243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1809799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1810361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1810890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1811442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1811944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1812462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1813211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1814031Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1815009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1815819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1816239Z ok (1.819s)
2022-11-23T01:42:03.1816743Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1460
2022-11-23T01:42:03.1817365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1461
2022-11-23T01:42:03.1817890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1462
2022-11-23T01:42:03.1818381Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1463
2022-11-23T01:42:03.1819084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1819571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1820205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1820707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1821174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1821844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1822314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1822982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1823522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1824016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1824790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1825293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1825974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1826619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1827124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1827842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1828318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1828993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1829545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1830167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1830716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1831307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1831893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1832449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1833264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1834071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1834850Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1835672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1836143Z ok (1.820s)
2022-11-23T01:42:03.1836613Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1516
2022-11-23T01:42:03.1837149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1517
2022-11-23T01:42:03.1837596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1518
2022-11-23T01:42:03.1838058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1519
2022-11-23T01:42:03.1838721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1839173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1839821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1840357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1840860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1841585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1842106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1842781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1843361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1843860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1844713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1845305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1846082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1846863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1847382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1848208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1848761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1849515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1850075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1850712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1851332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1851954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1852555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1853449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1854187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1855161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1856099Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1857002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1857490Z ok (1.819s)
2022-11-23T01:42:03.1858079Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1572
2022-11-23T01:42:03.1858765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1573
2022-11-23T01:42:03.1859364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1574
2022-11-23T01:42:03.1859955Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1575
2022-11-23T01:42:03.1860778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1861348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1862150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1862792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1863390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1864387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1864963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1865710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1866291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1866863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1867729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1868330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1869084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1869825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1870397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1871241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1871802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1872567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1873200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1873848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1874515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1875171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1875819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1876456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1877346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1878275Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1879223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1880175Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1880692Z ok (1.920s)
2022-11-23T01:42:03.1881243Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1628
2022-11-23T01:42:03.1881903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1629
2022-11-23T01:42:03.1882513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1630
2022-11-23T01:42:03.1883092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1631
2022-11-23T01:42:03.1883897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1884481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1885275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1885886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1886529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1887369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1887936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1888704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1889315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1889874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1890697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1891240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1891974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1892549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1893175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1894012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1894572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1895760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1896349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1897004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1897651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1898302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1898968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1899873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1900573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1901456Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1902412Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1903348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1903844Z ok (1.820s)
2022-11-23T01:42:03.1904371Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1684
2022-11-23T01:42:03.1905034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1685
2022-11-23T01:42:03.1905625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1686
2022-11-23T01:42:03.1906195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1687
2022-11-23T01:42:03.1907020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1907592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1908353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1908982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1909542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1910397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1910977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1911821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1912435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1913016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1913878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1914460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1915249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1915854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1916537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1917379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1917967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1918755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1919338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1919969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1920621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1921282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1921897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1922540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1923435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1924361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1925286Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1926323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1926851Z ok (9.637s)
2022-11-23T01:42:03.1927378Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1776
2022-11-23T01:42:03.1928048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1777
2022-11-23T01:42:03.1928620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1778
2022-11-23T01:42:03.1929203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1779
2022-11-23T01:42:03.1930017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1930603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1931384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1931980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1932548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1933414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1934019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1934758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1935513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1936080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1936889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1937455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1938206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1938801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1939484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1940342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1940924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1941696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1942264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1942933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1943584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1944238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1944922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1945584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1946489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1947437Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1948406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1949374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1950470Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:42:03.1951193Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:42:03.1952208Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:42:03.1952969Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:42:03.1953985Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:42:03.1954733Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:42:03.1955776Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:428: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2022-11-23T01:42:03.1956537Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2022-11-23T01:42:03.1956962Z ok (1.820s)
2022-11-23T01:42:03.1957463Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1832
2022-11-23T01:42:03.1958128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1833
2022-11-23T01:42:03.1958701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1834
2022-11-23T01:42:03.1959269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1835
2022-11-23T01:42:03.1960082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1960653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1961411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1962043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1962595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1963455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1964024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1964760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1965360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1966000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1966888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1967422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1968154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1968750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1969316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1970162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1970712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1971465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1972083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1972637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1973277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1973928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1974576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1975354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1976284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1977186Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1978100Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1979039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.1979556Z ok (1.820s)
2022-11-23T01:42:03.1980097Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1888
2022-11-23T01:42:03.1980775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1889
2022-11-23T01:42:03.1981355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1890
2022-11-23T01:42:03.1981922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1891
2022-11-23T01:42:03.1982751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1983341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1984134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1984843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1985430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.1986265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1986823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1987604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1988230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1988908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.1989762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1990350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1991137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1991779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1992336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.1993162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.1993741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.1994493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.1995124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.1995688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.1996315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.1996935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.1997594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.1998246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.1999161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2000120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2001090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2002025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2002526Z ok (1.820s)
2022-11-23T01:42:03.2003089Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1944
2022-11-23T01:42:03.2003787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1945
2022-11-23T01:42:03.2004396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1946
2022-11-23T01:42:03.2004989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1947
2022-11-23T01:42:03.2005830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2006474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2007228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2007845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2008508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.2009351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2009898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2010678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2011281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2011918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.2012763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2013354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2014132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2014696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2015380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.2016186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2016754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2017516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2018127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2018683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.2019336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.2020005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.2020644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.2021291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.2022211Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2023140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2024091Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2025006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2025494Z ok (1.820s)
2022-11-23T01:42:03.2026021Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2000
2022-11-23T01:42:03.2026686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2001
2022-11-23T01:42:03.2027256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2002
2022-11-23T01:42:03.2027840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2003
2022-11-23T01:42:03.2028648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2029242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2029960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2030554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2031276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.2032117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2032701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2033503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2034128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2034809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.2035635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2036204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2036962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2037543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2038112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.2038947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2039502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2040288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2040898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2041464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.2042097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.2042785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.2043349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.2043915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2044301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.2044794Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2045309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2045815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2046097Z ok (1.820s)
2022-11-23T01:42:03.2046537Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2056
2022-11-23T01:42:03.2047132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2057
2022-11-23T01:42:03.2047612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2058
2022-11-23T01:42:03.2047941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2059
2022-11-23T01:42:03.2048405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2048743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2049163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2049513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2049915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.2050365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2050697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2051123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2051464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2051812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.2052270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2052595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2053007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2053348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2053672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.2054128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2054439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2054998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2055359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2055667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.2056026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.2056403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.2056770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.2057257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2057650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.2058484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2059423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2060356Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2060865Z ok (1.820s)
2022-11-23T01:42:03.2061454Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2112
2022-11-23T01:42:03.2062153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2113
2022-11-23T01:42:03.2062735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2114
2022-11-23T01:42:03.2063319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2115
2022-11-23T01:42:03.2064138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2064699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2065471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2066096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2066794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:03.2067614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2068209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2068996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2069613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2070269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:03.2071122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2071725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2072491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2073116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2073686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:03.2074509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:03.2075089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:03.2075874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:03.2076490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:03.2077053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:03.2077721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:03.2078396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:03.2079054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:03.2079948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2080673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:03.2081566Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2082522Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2083422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:03.2083920Z ok (1.820s)
2022-11-23T01:42:03.2084104Z 
2022-11-23T01:42:03.2084442Z ----------------------------------------------------------------------
2022-11-23T01:42:03.2084849Z Ran 16 tests in 37.381s
2022-11-23T01:42:03.2085053Z 
2022-11-23T01:42:03.2085156Z OK
2022-11-23T01:42:03.2085311Z 
2022-11-23T01:42:03.2085452Z Generating XML reports...
2022-11-23T01:42:03.2086323Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123014125.xml
2022-11-23T01:42:03.2086750Z 
2022-11-23T01:42:03.2087248Z ##[endgroup]
2022-11-23T01:42:03.2088050Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_bz9tht5j)
2022-11-23T01:42:03.2088493Z 
2022-11-23T01:42:03.2088819Z Running distributed/fsdp/test_wrap ... [2022-11-23 01:42:03.178357]
2022-11-23T01:42:03.2089791Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:42:03.178699]
2022-11-23T01:42:45.5469850Z 
2022-11-23T01:42:45.5470439Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-11-23T01:42:45.5512598Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_nc55kxtn)
2022-11-23T01:42:45.5513499Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-11-23T01:42:45.5513861Z 
2022-11-23T01:42:45.5514002Z Running tests...
2022-11-23T01:42:45.5514608Z ----------------------------------------------------------------------
2022-11-23T01:42:45.5515384Z   test_always_wrap (__main__.TestAutoWrap)
2022-11-23T01:42:45.5515785Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5516454Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5517191Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5517642Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-11-23T01:42:45.5517962Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.000s)
2022-11-23T01:42:45.5518312Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-11-23T01:42:45.5518680Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5519062Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-11-23T01:42:45.5519422Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5519767Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-11-23T01:42:45.5520238Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5520570Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-11-23T01:42:45.5520964Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5523642Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5524469Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-11-23T01:42:45.5525045Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5525593Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-11-23T01:42:45.5526188Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5526722Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5527235Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-11-23T01:42:45.5527890Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-11-23T01:42:45.5528360Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5528787Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5529119Z   test_module_wrap_policy (__main__.TestAutoWrap)
2022-11-23T01:42:45.5529386Z Tests the ``ModuleWrapPolicy``. ... skip: Requires at least 2 GPUs (0.000s)
2022-11-23T01:42:45.5529722Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-11-23T01:42:45.5530022Z Tests the ``transformer_auto_wrap_policy``. ... skip: Requires at least 2 GPUs (0.000s)
2022-11-23T01:42:45.5530352Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5530707Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5531068Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5531442Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-11-23T01:42:45.5531752Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-11-23T01:42:45.5532140Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2179
2022-11-23T01:42:45.5532541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2180
2022-11-23T01:42:45.5532856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2181
2022-11-23T01:42:45.5535559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2182
2022-11-23T01:42:45.5536259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5536596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5537047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5537396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5537721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5538196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5538528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5538944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5539290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5539630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5540100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5540413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5540839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5541184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5541521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5541969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5542297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5542834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5543164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5543497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5555063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5555966Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5557140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5557699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5558090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5558427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5558772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5559115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5559435Z dist init r=1, world=4
2022-11-23T01:42:45.5559734Z dist init r=0, world=4
2022-11-23T01:42:45.5560043Z dist init r=3, world=4
2022-11-23T01:42:45.5560336Z dist init r=2, world=4
2022-11-23T01:42:45.5560692Z skip: Need at least 2 CUDA devices (2.077s)
2022-11-23T01:42:45.5561137Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T01:42:45.5561922Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2235
2022-11-23T01:42:45.5562571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2236
2022-11-23T01:42:45.5563098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2237
2022-11-23T01:42:45.5563632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2238
2022-11-23T01:42:45.5564375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5564891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5565601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5566238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5566780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5567637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5568170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5568867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5569481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5569967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5570437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5570771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5571460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5572025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5572624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5573509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5574413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5575163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5575500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5576020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5576356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5576695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5577192Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5578010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5578378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5578725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5579066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5579400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5579659Z dist init r=0, world=4
2022-11-23T01:42:45.5579843Z dist init r=1, world=4
2022-11-23T01:42:45.5580021Z dist init r=3, world=4
2022-11-23T01:42:45.5580183Z dist init r=2, world=4
2022-11-23T01:42:45.5580393Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:42:45.5580709Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T01:42:45.5581124Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2291
2022-11-23T01:42:45.5581517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2292
2022-11-23T01:42:45.5581840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2293
2022-11-23T01:42:45.5582147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2294
2022-11-23T01:42:45.5582601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5582929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5583361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5583695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5584032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5584499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5584828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5585243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5585584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5585914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5586440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5586775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5587207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5587551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5587871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5588367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5588918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5589410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5589726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5590152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5590497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5590816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5591309Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5591824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5592212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5592546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5592885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5593229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5593486Z dist init r=3, world=4
2022-11-23T01:42:45.5593654Z dist init r=0, world=4
2022-11-23T01:42:45.5593830Z dist init r=2, world=4
2022-11-23T01:42:45.5594005Z dist init r=1, world=4
2022-11-23T01:42:45.5594195Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:42:45.5594504Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T01:42:45.5594929Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2347
2022-11-23T01:42:45.5595315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2348
2022-11-23T01:42:45.5595639Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2349
2022-11-23T01:42:45.5595965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2350
2022-11-23T01:42:45.5596415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5596731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5597158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5597503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5597823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5598292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5598616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5599069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5599402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5599734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5600197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5600519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5600927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5601300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5601633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5602087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5602417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5602846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5603188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5603505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5603995Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5604515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5605032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5605535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5605921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5606360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5606692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5607036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5607292Z dist init r=3, world=4
2022-11-23T01:42:45.5607477Z dist init r=0, world=4
2022-11-23T01:42:45.5607642Z dist init r=2, world=4
2022-11-23T01:42:45.5607818Z dist init r=1, world=4
2022-11-23T01:42:45.5608024Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:42:45.5608322Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T01:42:45.5608751Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2403
2022-11-23T01:42:45.5609147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2404
2022-11-23T01:42:45.5609458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2405
2022-11-23T01:42:45.5609778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2406
2022-11-23T01:42:45.5610231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5610560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5610981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5611323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5611690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5612158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5612471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5612892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5613234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5613554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5614050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5614370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5614793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5615259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5615592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5616056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5616367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5616791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5617137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5617467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5617945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5618473Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5618985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5619498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5619866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5620212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5620559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5620903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5621147Z dist init r=3, world=4
2022-11-23T01:42:45.5621328Z dist init r=1, world=4
2022-11-23T01:42:45.5621511Z dist init r=2, world=4
2022-11-23T01:42:45.5621673Z dist init r=0, world=4
2022-11-23T01:42:45.5621877Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:42:45.5622389Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2459
2022-11-23T01:42:45.5622996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2460
2022-11-23T01:42:45.5623324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2461
2022-11-23T01:42:45.5623632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2462
2022-11-23T01:42:45.5624084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5624482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5624901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5625247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5625584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5626052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5626366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5626839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5627181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5627501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5627966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5628289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5628715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5629040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5629375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5629847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5630171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5630582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5630924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5631256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5631733Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5632254Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5632771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5633290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5633662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5634011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5634360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5634698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5634943Z dist init r=3, world=4
2022-11-23T01:42:45.5635123Z dist init r=0, world=4
2022-11-23T01:42:45.5635300Z dist init r=2, world=4
2022-11-23T01:42:45.5635461Z dist init r=1, world=4
2022-11-23T01:42:45.5635665Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5636175Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2515
2022-11-23T01:42:45.5636708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2516
2022-11-23T01:42:45.5637055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2517
2022-11-23T01:42:45.5637373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2518
2022-11-23T01:42:45.5637823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5638150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5638566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5638908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5639275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5639731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5640059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5640489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5640832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5641156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5641621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5641950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5642379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5642707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5643041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5643536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5644019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5644342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5644766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5645106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5645425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5645916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5646508Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5647021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5647391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5647737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5648078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5648406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5648666Z dist init r=2, world=4
2022-11-23T01:42:45.5648847Z dist init r=3, world=4
2022-11-23T01:42:45.5649009Z dist init r=0, world=4
2022-11-23T01:42:45.5649181Z dist init r=1, world=4
2022-11-23T01:42:45.5649385Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5649926Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2571
2022-11-23T01:42:45.5650452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2572
2022-11-23T01:42:45.5650779Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2573
2022-11-23T01:42:45.5651103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2574
2022-11-23T01:42:45.5651551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5651897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5652321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5652663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5652982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5653444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5653771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5654191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5654519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5654965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5655678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5656146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5656581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5656923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5657251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5657698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5658025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5658449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5658790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5659106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5659599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5660116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5660632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5661130Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5661514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5661863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5662190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5662529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5662845Z dist init r=3, world=4
2022-11-23T01:42:45.5663028Z dist init r=1, world=4
2022-11-23T01:42:45.5663190Z dist init r=0, world=4
2022-11-23T01:42:45.5663363Z dist init r=2, world=4
2022-11-23T01:42:45.5663565Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5664060Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2627
2022-11-23T01:42:45.5664632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2628
2022-11-23T01:42:45.5664957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2629
2022-11-23T01:42:45.5665276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2630
2022-11-23T01:42:45.5665714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5666044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5666471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5666803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5667138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5667608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5667935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5668348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5668688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5669023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5669474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5669794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5670216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5670552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5670871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5671336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5671659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5672081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5672405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5672733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5673224Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5673728Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5674243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5674754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5675194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5675528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5675871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5676204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5676460Z dist init r=1, world=4
2022-11-23T01:42:45.5676628Z dist init r=2, world=4
2022-11-23T01:42:45.5676800Z dist init r=3, world=4
2022-11-23T01:42:45.5676974Z dist init r=0, world=4
2022-11-23T01:42:45.5677194Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:42:45.5677698Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2683
2022-11-23T01:42:45.5678222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2684
2022-11-23T01:42:45.5678547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2685
2022-11-23T01:42:45.5678856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2686
2022-11-23T01:42:45.5679301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5679629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5680040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5680387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5680720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5681187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5681496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5681921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5682263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5682583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5683049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5683374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5683796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5684126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5684459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5684924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5685246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5685656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5686074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5686408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5686890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5687410Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5687960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5688474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5688840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5689185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5689525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5689900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5690149Z dist init r=0, world=4
2022-11-23T01:42:45.5690329Z dist init r=1, world=4
2022-11-23T01:42:45.5690506Z dist init r=3, world=4
2022-11-23T01:42:45.5690669Z dist init r=2, world=4
2022-11-23T01:42:45.5690873Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5691386Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2739
2022-11-23T01:42:45.5691915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2740
2022-11-23T01:42:45.5692227Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2741
2022-11-23T01:42:45.5692543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2742
2022-11-23T01:42:45.5692995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5693308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5693738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5694085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5694420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5694985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5695326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5695757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5696087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5696419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5696885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5697216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5697626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5697967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5698297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5698790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5699275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5699599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5700026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5700417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5700752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5701242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5701758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5702258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5702679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5703023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5703365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5703695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5703948Z dist init r=2, world=4
2022-11-23T01:42:45.5704129Z dist init r=0, world=4
2022-11-23T01:42:45.5704291Z dist init r=1, world=4
2022-11-23T01:42:45.5704465Z dist init r=3, world=4
2022-11-23T01:42:45.5704668Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5705160Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2795
2022-11-23T01:42:45.5705691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2796
2022-11-23T01:42:45.5706015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2797
2022-11-23T01:42:45.5706338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2798
2022-11-23T01:42:45.5706776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5707104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5707528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5707873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5708195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5708664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5708989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5709399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5709743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5710075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5710561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5711043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5711367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5711792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5712132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5712450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5712971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5713466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5713773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5714193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5714534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5714896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5715370Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5715884Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5716272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5716620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5716951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5717284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5717542Z dist init r=1, world=4
2022-11-23T01:42:45.5717708Z dist init r=2, world=4
2022-11-23T01:42:45.5717887Z dist init r=3, world=4
2022-11-23T01:42:45.5718060Z dist init r=0, world=4
2022-11-23T01:42:45.5718249Z skip: Need at least 2 CUDA devices (1.921s)
2022-11-23T01:42:45.5718755Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2851
2022-11-23T01:42:45.5719287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2852
2022-11-23T01:42:45.5719613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2853
2022-11-23T01:42:45.5719921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2854
2022-11-23T01:42:45.5720363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5720691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5721118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5721450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5721786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5722254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5722564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5722987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5723330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5723663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5724144Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5724640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5724964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5725417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5725749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5726160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5726627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5726940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5727360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5727735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5728066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5728547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5729062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5729574Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5729954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5730288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5730628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5730886Z dist init r=0, world=4
2022-11-23T01:42:45.5731053Z dist init r=1, world=4
2022-11-23T01:42:45.5731227Z dist init r=2, world=4
2022-11-23T01:42:45.5731490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5731733Z dist init r=3, world=4
2022-11-23T01:42:45.5731938Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5732440Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2907
2022-11-23T01:42:45.5732971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2908
2022-11-23T01:42:45.5733281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2909
2022-11-23T01:42:45.5733604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2910
2022-11-23T01:42:45.5734054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5734381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5734797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5735266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5735603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5736063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5736385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5736812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5737149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5737465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5737984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5738310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5738718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5739056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5739387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5739850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5740199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5740621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5740962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5741294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5741772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5742289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5742804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5743315Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5743684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5744034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5744378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5744707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5744963Z dist init r=1, world=4
2022-11-23T01:42:45.5745143Z dist init r=0, world=4
2022-11-23T01:42:45.5745306Z dist init r=3, world=4
2022-11-23T01:42:45.5745480Z dist init r=2, world=4
2022-11-23T01:42:45.5745682Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:42:45.5746187Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2963
2022-11-23T01:42:45.5746707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2964
2022-11-23T01:42:45.5747035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2965
2022-11-23T01:42:45.5747358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2966
2022-11-23T01:42:45.5747810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5748122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5748547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5748891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5749215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5749685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5750010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5750462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5750792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5751123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5751589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5751901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5752327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5752695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5753025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5753476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5753802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5754225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5754561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5754878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5755373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5755892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5756396Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5756907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5757291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5757636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5757966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5758308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5758561Z dist init r=1, world=4
2022-11-23T01:42:45.5758738Z dist init r=3, world=4
2022-11-23T01:42:45.5758901Z dist init r=0, world=4
2022-11-23T01:42:45.5759075Z dist init r=2, world=4
2022-11-23T01:42:45.5759279Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5759774Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3019
2022-11-23T01:42:45.5760301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3020
2022-11-23T01:42:45.5760626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3021
2022-11-23T01:42:45.5760948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3022
2022-11-23T01:42:45.5761385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5761717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5762145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5762475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5762863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5763332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5763658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5764070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5764411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5764742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5765222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5765545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5765971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5766398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5766717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5767183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5767511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5767935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5768266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5768596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5769087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5769592Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5770105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5770619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5771001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5771336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5771675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5772017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5772273Z dist init r=2, world=4
2022-11-23T01:42:45.5772441Z dist init r=3, world=4
2022-11-23T01:42:45.5772617Z dist init r=1, world=4
2022-11-23T01:42:45.5772789Z dist init r=0, world=4
2022-11-23T01:42:45.5772981Z skip: Need at least 2 CUDA devices (1.822s)
2022-11-23T01:42:45.5773484Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3075
2022-11-23T01:42:45.5774019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3076
2022-11-23T01:42:45.5774347Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3077
2022-11-23T01:42:45.5774653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3078
2022-11-23T01:42:45.5775233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5775622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5776040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5776383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5776715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5777181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5777536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5777959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5778299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5778618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5779081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5779405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5779826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5780153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5780481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5780976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5781474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5781787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5782213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5782555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5782872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5783367Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5783886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5784407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5784775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5785120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5785464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5785800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5786042Z dist init r=3, world=4
2022-11-23T01:42:45.5786219Z dist init r=1, world=4
2022-11-23T01:42:45.5786398Z dist init r=2, world=4
2022-11-23T01:42:45.5786559Z dist init r=0, world=4
2022-11-23T01:42:45.5786761Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:42:45.5787267Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3131
2022-11-23T01:42:45.5787791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3132
2022-11-23T01:42:45.5788141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3133
2022-11-23T01:42:45.5788465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3134
2022-11-23T01:42:45.5788909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5789222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5789646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5790028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5790362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5790816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5791140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5791605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5791951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5792271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5792768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5793420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5793838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5794430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5794958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5795489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5796221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5796748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5797444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5798021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5798565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5799322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5800111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5800639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5801030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5801369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5801710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5802057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5802320Z dist init r=0, world=4
2022-11-23T01:42:45.5802492Z dist init r=3, world=4
2022-11-23T01:42:45.5802674Z dist init r=2, world=4
2022-11-23T01:42:45.5802852Z dist init r=1, world=4
2022-11-23T01:42:45.5803045Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5803648Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3187
2022-11-23T01:42:45.5804192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3188
2022-11-23T01:42:45.5804508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3189
2022-11-23T01:42:45.5804831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3190
2022-11-23T01:42:45.5805325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5805656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5822413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5822870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5823217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5823739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5824078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5824503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5824873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5825218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5825677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5826009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5826440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5826787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5827110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5827583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5827921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5828362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5828713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5829030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5829531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5830052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5830564Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5831068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5831454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5831797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5832127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5832577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5832835Z dist init r=1, world=4
2022-11-23T01:42:45.5833016Z dist init r=2, world=4
2022-11-23T01:42:45.5833180Z dist init r=3, world=4
2022-11-23T01:42:45.5833355Z dist init r=0, world=4
2022-11-23T01:42:45.5833561Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:42:45.5834056Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3243
2022-11-23T01:42:45.5834631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3244
2022-11-23T01:42:45.5834961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3245
2022-11-23T01:42:45.5835287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3246
2022-11-23T01:42:45.5835735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5836066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5836497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5836830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5837168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5837636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5837965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5838379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5838727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5839061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5839525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5839837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5840261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5840617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5841090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5841829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5842619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5843147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5843651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5843989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5844330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5844830Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5845357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5845864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5846466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5846822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5847152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5847498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5847752Z dist init r=2, world=4
2022-11-23T01:42:45.5847933Z dist init r=0, world=4
2022-11-23T01:42:45.5848098Z dist init r=1, world=4
2022-11-23T01:42:45.5848277Z dist init r=3, world=4
2022-11-23T01:42:45.5848517Z skip: Need at least 2 CUDA devices (1.821s)
2022-11-23T01:42:45.5849012Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3299
2022-11-23T01:42:45.5849554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3300
2022-11-23T01:42:45.5849886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3301
2022-11-23T01:42:45.5850215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3302
2022-11-23T01:42:45.5850660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5850989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5851421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5851757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5852094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5852589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5852934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5853368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5853709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5854068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5854833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5855330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5855758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5856103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5856454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5856938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5857255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5857856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5858417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5858966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5859710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5860243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5860903Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5861411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5861801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5862150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5862497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5862875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5863130Z dist init r=3, world=4
2022-11-23T01:42:45.5863313Z dist init r=2, world=4
2022-11-23T01:42:45.5863479Z dist init r=1, world=4
2022-11-23T01:42:45.5863656Z dist init r=0, world=4
2022-11-23T01:42:45.5863864Z skip: Need at least 2 CUDA devices (1.721s)
2022-11-23T01:42:45.5864224Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3355
2022-11-23T01:42:45.5864633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3356
2022-11-23T01:42:45.5864963Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3357
2022-11-23T01:42:45.5865287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3358
2022-11-23T01:42:45.5865725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5866057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5866484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5866833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5867155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5867622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5867949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5868360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5868702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5869041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5869508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5869820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5870247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5870592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5870928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5871408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5871925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5872420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5872737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5873193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5873539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5873874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5874352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5874869Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5875255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5875640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5875972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5876314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5876574Z dist init r=1, world=4
2022-11-23T01:42:45.5876740Z dist init r=0, world=4
2022-11-23T01:42:45.5876920Z dist init r=2, world=4
2022-11-23T01:42:45.5877097Z dist init r=3, world=4
2022-11-23T01:42:45.5877289Z skip: Need at least 2 CUDA devices (1.719s)
2022-11-23T01:42:45.5877660Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3411
2022-11-23T01:42:45.5878064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3412
2022-11-23T01:42:45.5878393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3413
2022-11-23T01:42:45.5878705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3414
2022-11-23T01:42:45.5879155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5879486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5879899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5880244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5880581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:42:45.5881053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5881366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5881793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5882138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5882475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:42:45.5882929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5883256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5883683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5884011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5884344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T01:42:45.5884810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:42:45.5885135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:42:45.5885549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:42:45.5885923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:42:45.5886361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T01:42:45.5886842Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5887364Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5887874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5888427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T01:42:45.5888795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T01:42:45.5889141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:42:45.5889481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:42:45.5889829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T01:42:45.5890072Z dist init r=2, world=4
2022-11-23T01:42:45.5890253Z dist init r=0, world=4
2022-11-23T01:42:45.5890431Z dist init r=1, world=4
2022-11-23T01:42:45.5890592Z dist init r=3, world=4
2022-11-23T01:42:45.5890797Z skip: Need at least 2 CUDA devices (1.720s)
2022-11-23T01:42:45.5890935Z 
2022-11-23T01:42:45.5891137Z ----------------------------------------------------------------------
2022-11-23T01:42:45.5891365Z Ran 47 tests in 40.760s
2022-11-23T01:42:45.5891478Z 
2022-11-23T01:42:45.5891551Z OK (skipped=47)
2022-11-23T01:42:45.5891659Z 
2022-11-23T01:42:45.5891743Z Generating XML reports...
2022-11-23T01:42:45.5892175Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123014204.xml
2022-11-23T01:42:45.5892705Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123014204.xml
2022-11-23T01:42:45.5892944Z 
2022-11-23T01:42:45.5893267Z ##[endgroup]
2022-11-23T01:42:45.5893693Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_nc55kxtn)
2022-11-23T01:42:45.5893939Z 
2022-11-23T01:42:45.5894110Z Running distributed/test_c10d_nccl ... [2022-11-23 01:42:45.548369]
2022-11-23T01:42:45.5894630Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:42:45.548715]
2022-11-23T01:46:33.5921076Z 
2022-11-23T01:46:33.5921598Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2022-11-23T01:46:33.5924481Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_7dwezvtx)
2022-11-23T01:46:33.5927045Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-11-23T01:46:33.5929470Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-11-23T01:46:33.5929823Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-11-23T01:46:33.5930196Z test_nccl_barrier (__main__.CommTest)
2022-11-23T01:46:33.5933426Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-11-23T01:46:33.5934059Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-11-23T01:46:33.5934479Z test_nccl_barrier_timeout (__main__.CommTest)
2022-11-23T01:46:33.5935062Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-11-23T01:46:33.5935475Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-11-23T01:46:33.5935927Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-11-23T01:46:33.5936347Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-11-23T01:46:33.5936775Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-11-23T01:46:33.5937145Z test_nncl_rank_membership (__main__.CommTest)
2022-11-23T01:46:33.5937571Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-11-23T01:46:33.5938038Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-11-23T01:46:33.5938466Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-11-23T01:46:33.5976892Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-11-23T01:46:33.5977327Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-11-23T01:46:33.5977728Z test_tensor_dtype_complex (__main__.CommTest)
2022-11-23T01:46:33.5978081Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-11-23T01:46:33.5979088Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-11-23T01:46:33.5980116Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.5980473Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.5980883Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.5981303Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.5981727Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-11-23T01:46:33.5982125Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.5982525Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-11-23T01:46:33.6015972Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-11-23T01:46:33.6024781Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6025560Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6025927Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6026261Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6026602Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6026927Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6027234Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6027571Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6027897Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6028220Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6028557Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6028919Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6029400Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6029869Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6030606Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6031206Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6031789Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6032350Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6032946Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6033579Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6034192Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6034821Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6035397Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6035990Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6036569Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6037140Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6037706Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6038342Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6038840Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6039336Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6039850Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6040362Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6040902Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6041420Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6042033Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6042588Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6043161Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6043763Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6044361Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6044968Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6045483Z test_fp16 (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6046084Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6046625Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6046936Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6047270Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6047602Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6047918Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6048245Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6048596Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6048946Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6049315Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6049810Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6050391Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6050739Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6051091Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6051449Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6051801Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6052142Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6052436Z test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6052722Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6053028Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6053332Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6053667Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6053983Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6054304Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6054579Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:46:33.6055785Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-11-23T01:46:33.6056789Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6057095Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6057379Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6057678Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6057977Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6058261Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6058552Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6058844Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6059098Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-11-23T01:46:33.6059892Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2022-11-23T01:46:33.6060731Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T01:46:33.6061132Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T01:46:33.6061525Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T01:46:33.6061909Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2022-11-23T01:46:33.6062313Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-11-23T01:46:33.6062654Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-11-23T01:46:33.6064221Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2022-11-23T01:46:33.6065827Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6066137Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6066407Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6066653Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6066908Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6067165Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6067422Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6067664Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6067924Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6068179Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6068465Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6068732Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6069003Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6069274Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6069552Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6069819Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6070077Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6070322Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6070577Z test_send_recv (__main__.ProcessGroupNCCLTest)
2022-11-23T01:46:33.6070886Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-11-23T01:46:33.6071175Z test_common_errors (__main__.RendezvousEnvTest)
2022-11-23T01:46:33.6071416Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:46:33.6071724Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-11-23T01:46:33.6072022Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-11-23T01:46:33.6072494Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6072967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6073302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6073722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6074072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6074237Z 
2022-11-23T01:46:33.6074313Z Running tests...
2022-11-23T01:46:33.6074604Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6074955Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6075156Z 
2022-11-23T01:46:33.6075350Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6075589Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6075687Z 
2022-11-23T01:46:33.6075759Z OK (skipped=1)
2022-11-23T01:46:33.6075867Z 
2022-11-23T01:46:33.6075954Z Generating XML reports...
2022-11-23T01:46:33.6076364Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014248.xml
2022-11-23T01:46:33.6076802Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6077258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6077586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6078010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6078344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6078507Z 
2022-11-23T01:46:33.6078580Z Running tests...
2022-11-23T01:46:33.6078879Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6079244Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6079446Z 
2022-11-23T01:46:33.6079639Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6079877Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6079986Z 
2022-11-23T01:46:33.6080056Z OK (skipped=1)
2022-11-23T01:46:33.6080162Z 
2022-11-23T01:46:33.6080231Z Generating XML reports...
2022-11-23T01:46:33.6080636Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014249.xml
2022-11-23T01:46:33.6081112Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6081571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6081883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6082312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6082657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6082821Z 
2022-11-23T01:46:33.6082880Z Running tests...
2022-11-23T01:46:33.6083176Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6083505Z   test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6083694Z 
2022-11-23T01:46:33.6083890Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6084114Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6084224Z 
2022-11-23T01:46:33.6084297Z OK (skipped=1)
2022-11-23T01:46:33.6084403Z 
2022-11-23T01:46:33.6084487Z Generating XML reports...
2022-11-23T01:46:33.6084875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014250.xml
2022-11-23T01:46:33.6085320Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6085872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6086199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6086612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6086955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6087119Z 
2022-11-23T01:46:33.6087192Z Running tests...
2022-11-23T01:46:33.6087483Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6087825Z   test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6088025Z 
2022-11-23T01:46:33.6088220Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6088460Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6088559Z 
2022-11-23T01:46:33.6088630Z OK (skipped=1)
2022-11-23T01:46:33.6088735Z 
2022-11-23T01:46:33.6088820Z Generating XML reports...
2022-11-23T01:46:33.6089223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014252.xml
2022-11-23T01:46:33.6089652Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6090114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6090442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6090870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6091201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6091363Z 
2022-11-23T01:46:33.6091440Z Running tests...
2022-11-23T01:46:33.6091778Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6092125Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6092337Z 
2022-11-23T01:46:33.6092527Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6093688Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6093800Z 
2022-11-23T01:46:33.6093872Z OK (skipped=1)
2022-11-23T01:46:33.6093979Z 
2022-11-23T01:46:33.6094050Z Generating XML reports...
2022-11-23T01:46:33.6094451Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014253.xml
2022-11-23T01:46:33.6095115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6095589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6095909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6096337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6096686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6096851Z 
2022-11-23T01:46:33.6096911Z Running tests...
2022-11-23T01:46:33.6097218Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6097556Z   test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6097751Z 
2022-11-23T01:46:33.6097942Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6098168Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6098277Z 
2022-11-23T01:46:33.6098349Z OK (skipped=1)
2022-11-23T01:46:33.6098455Z 
2022-11-23T01:46:33.6098540Z Generating XML reports...
2022-11-23T01:46:33.6098932Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014255.xml
2022-11-23T01:46:33.6099384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6099841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6100172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6100587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6100929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6101094Z 
2022-11-23T01:46:33.6101167Z Running tests...
2022-11-23T01:46:33.6101466Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6101802Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6102006Z 
2022-11-23T01:46:33.6102201Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6102439Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6102550Z 
2022-11-23T01:46:33.6102608Z OK (skipped=1)
2022-11-23T01:46:33.6102713Z 
2022-11-23T01:46:33.6102796Z Generating XML reports...
2022-11-23T01:46:33.6103196Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014256.xml
2022-11-23T01:46:33.6103642Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6104088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6104418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6104842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6105238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6105402Z 
2022-11-23T01:46:33.6105476Z Running tests...
2022-11-23T01:46:33.6105775Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6106135Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6106367Z 
2022-11-23T01:46:33.6106548Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6106793Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6106905Z 
2022-11-23T01:46:33.6106977Z OK (skipped=1)
2022-11-23T01:46:33.6107127Z 
2022-11-23T01:46:33.6107198Z Generating XML reports...
2022-11-23T01:46:33.6107604Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014258.xml
2022-11-23T01:46:33.6108048Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6108509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6108824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6109248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6109596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6109763Z 
2022-11-23T01:46:33.6109835Z Running tests...
2022-11-23T01:46:33.6110119Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6110474Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6110679Z 
2022-11-23T01:46:33.6110870Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6111093Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6111206Z 
2022-11-23T01:46:33.6111279Z OK (skipped=1)
2022-11-23T01:46:33.6111386Z 
2022-11-23T01:46:33.6111472Z Generating XML reports...
2022-11-23T01:46:33.6111872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014259.xml
2022-11-23T01:46:33.6112306Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6112762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6113095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6113509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6113856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6114018Z 
2022-11-23T01:46:33.6114091Z Running tests...
2022-11-23T01:46:33.6114389Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6114722Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6114925Z 
2022-11-23T01:46:33.6115118Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6115359Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6115468Z 
2022-11-23T01:46:33.6115526Z OK (skipped=1)
2022-11-23T01:46:33.6115632Z 
2022-11-23T01:46:33.6115716Z Generating XML reports...
2022-11-23T01:46:33.6116116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014301.xml
2022-11-23T01:46:33.6116563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6117011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6117340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6117804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6118137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6118300Z 
2022-11-23T01:46:33.6118372Z Running tests...
2022-11-23T01:46:33.6118669Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6119016Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6119218Z 
2022-11-23T01:46:33.6119397Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6119667Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6119779Z 
2022-11-23T01:46:33.6119851Z OK (skipped=1)
2022-11-23T01:46:33.6119956Z 
2022-11-23T01:46:33.6120039Z Generating XML reports...
2022-11-23T01:46:33.6120429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014302.xml
2022-11-23T01:46:33.6120885Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6121350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6121664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6122089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6122433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6122597Z 
2022-11-23T01:46:33.6122673Z Running tests...
2022-11-23T01:46:33.6122959Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6123293Z   test_nncl_rank_membership (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6123491Z 
2022-11-23T01:46:33.6123684Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6123913Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6124024Z 
2022-11-23T01:46:33.6124096Z OK (skipped=1)
2022-11-23T01:46:33.6124203Z 
2022-11-23T01:46:33.6124289Z Generating XML reports...
2022-11-23T01:46:33.6124693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014303.xml
2022-11-23T01:46:33.6125127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6125655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6125992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6126410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6126755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6126919Z 
2022-11-23T01:46:33.6126991Z Running tests...
2022-11-23T01:46:33.6127291Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6127636Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6127847Z 
2022-11-23T01:46:33.6128040Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6128276Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6128386Z 
2022-11-23T01:46:33.6128444Z OK (skipped=1)
2022-11-23T01:46:33.6128550Z 
2022-11-23T01:46:33.6128634Z Generating XML reports...
2022-11-23T01:46:33.6129034Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014305.xml
2022-11-23T01:46:33.6129484Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6129927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6130295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6130724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6131067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6131218Z 
2022-11-23T01:46:33.6131291Z Running tests...
2022-11-23T01:46:33.6131588Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6131949Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6132205Z 
2022-11-23T01:46:33.6132381Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6132618Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6132729Z 
2022-11-23T01:46:33.6132801Z OK (skipped=1)
2022-11-23T01:46:33.6132908Z 
2022-11-23T01:46:33.6132991Z Generating XML reports...
2022-11-23T01:46:33.6133382Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014306.xml
2022-11-23T01:46:33.6133830Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6134288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6134605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6135188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6135537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6135701Z 
2022-11-23T01:46:33.6135773Z Running tests...
2022-11-23T01:46:33.6136063Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6136421Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6136638Z 
2022-11-23T01:46:33.6136832Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6137055Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6137165Z 
2022-11-23T01:46:33.6137238Z OK (skipped=1)
2022-11-23T01:46:33.6137344Z 
2022-11-23T01:46:33.6137428Z Generating XML reports...
2022-11-23T01:46:33.6137833Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014308.xml
2022-11-23T01:46:33.6138267Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6138725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6139056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6139485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6139818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6139981Z 
2022-11-23T01:46:33.6140054Z Running tests...
2022-11-23T01:46:33.6140357Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6140692Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6140896Z 
2022-11-23T01:46:33.6141088Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6141325Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6141438Z 
2022-11-23T01:46:33.6141511Z OK (skipped=1)
2022-11-23T01:46:33.6141605Z 
2022-11-23T01:46:33.6141691Z Generating XML reports...
2022-11-23T01:46:33.6142093Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014309.xml
2022-11-23T01:46:33.6142538Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6143037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6143367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6143791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6144133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6144295Z 
2022-11-23T01:46:33.6144367Z Running tests...
2022-11-23T01:46:33.6144697Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6145063Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6145266Z 
2022-11-23T01:46:33.6145455Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6145690Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6145799Z 
2022-11-23T01:46:33.6145858Z OK (skipped=1)
2022-11-23T01:46:33.6145967Z 
2022-11-23T01:46:33.6146053Z Generating XML reports...
2022-11-23T01:46:33.6146453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014311.xml
2022-11-23T01:46:33.6146897Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6147343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6147673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6148096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6148430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6148594Z 
2022-11-23T01:46:33.6148666Z Running tests...
2022-11-23T01:46:33.6148967Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6149308Z   test_tensor_dtype_complex (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6149489Z 
2022-11-23T01:46:33.6149683Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6149920Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6150030Z 
2022-11-23T01:46:33.6150100Z OK (skipped=1)
2022-11-23T01:46:33.6150205Z 
2022-11-23T01:46:33.6150276Z Generating XML reports...
2022-11-23T01:46:33.6150674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014312.xml
2022-11-23T01:46:33.6151168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6151625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6151938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6152363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6152705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6152867Z 
2022-11-23T01:46:33.6152938Z Running tests...
2022-11-23T01:46:33.6153220Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6153557Z   test_tensor_dtype_mismatch (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6153753Z 
2022-11-23T01:46:33.6153943Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6154170Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6154278Z 
2022-11-23T01:46:33.6154348Z OK (skipped=1)
2022-11-23T01:46:33.6154454Z 
2022-11-23T01:46:33.6154537Z Generating XML reports...
2022-11-23T01:46:33.6154923Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014314.xml
2022-11-23T01:46:33.6155405Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6155865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6156196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6156604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6156948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6157111Z 
2022-11-23T01:46:33.6157181Z Running tests...
2022-11-23T01:46:33.6157507Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6157874Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3678
2022-11-23T01:46:33.6158256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3679
2022-11-23T01:46:33.6158703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6159018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6159444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6159784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6160105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6160550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6160873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6161296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6161641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6161947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6162237Z skip: Need at least 2 CUDA devices (1.768s)
2022-11-23T01:46:33.6162373Z 
2022-11-23T01:46:33.6162568Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6162787Z Ran 1 test in 1.768s
2022-11-23T01:46:33.6162897Z 
2022-11-23T01:46:33.6162966Z OK (skipped=1)
2022-11-23T01:46:33.6163075Z 
2022-11-23T01:46:33.6163158Z Generating XML reports...
2022-11-23T01:46:33.6163557Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014315.xml
2022-11-23T01:46:33.6164014Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6164473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6164798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6165208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6165619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6165788Z 
2022-11-23T01:46:33.6165860Z Running tests...
2022-11-23T01:46:33.6166159Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6166526Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3711
2022-11-23T01:46:33.6166909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3712
2022-11-23T01:46:33.6167359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6167671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6168129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6168471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6168792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6169239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6169566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6169991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6170353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6170672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6170958Z skip: Need at least 2 CUDA devices (1.870s)
2022-11-23T01:46:33.6171093Z 
2022-11-23T01:46:33.6171287Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6171509Z Ran 1 test in 1.871s
2022-11-23T01:46:33.6171618Z 
2022-11-23T01:46:33.6171689Z OK (skipped=1)
2022-11-23T01:46:33.6171793Z 
2022-11-23T01:46:33.6171877Z Generating XML reports...
2022-11-23T01:46:33.6172274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014318.xml
2022-11-23T01:46:33.6172734Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6173194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6173520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6173932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6174280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6174442Z 
2022-11-23T01:46:33.6174512Z Running tests...
2022-11-23T01:46:33.6174794Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6175349Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3744
2022-11-23T01:46:33.6175731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3745
2022-11-23T01:46:33.6176182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6176496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6176924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6177267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6177590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6178036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6178358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6178780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6179109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6179427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6179717Z skip: Need at least 2 CUDA devices (1.769s)
2022-11-23T01:46:33.6179851Z 
2022-11-23T01:46:33.6180046Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6180269Z Ran 1 test in 1.769s
2022-11-23T01:46:33.6180379Z 
2022-11-23T01:46:33.6180449Z OK (skipped=1)
2022-11-23T01:46:33.6180617Z 
2022-11-23T01:46:33.6180703Z Generating XML reports...
2022-11-23T01:46:33.6181106Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014322.xml
2022-11-23T01:46:33.6181561Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6182015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6182338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6182747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6183124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6183286Z 
2022-11-23T01:46:33.6183358Z Running tests...
2022-11-23T01:46:33.6183645Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6184033Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3777
2022-11-23T01:46:33.6184420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3778
2022-11-23T01:46:33.6184869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6185184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6185606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6185949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6186272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6186715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6187041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6187465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6187793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6188111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6188401Z skip: Need at least 2 CUDA devices (1.770s)
2022-11-23T01:46:33.6188534Z 
2022-11-23T01:46:33.6188730Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6188960Z Ran 1 test in 1.771s
2022-11-23T01:46:33.6189070Z 
2022-11-23T01:46:33.6189143Z OK (skipped=1)
2022-11-23T01:46:33.6189249Z 
2022-11-23T01:46:33.6189333Z Generating XML reports...
2022-11-23T01:46:33.6189733Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014325.xml
2022-11-23T01:46:33.6190194Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6190652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6190979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6191392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6191735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6191898Z 
2022-11-23T01:46:33.6191972Z Running tests...
2022-11-23T01:46:33.6192255Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6192633Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3810
2022-11-23T01:46:33.6193051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3811
2022-11-23T01:46:33.6193503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6193819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6194243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6194587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6194910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6195383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6195707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6196131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6196466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6196786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6197074Z skip: Need at least 2 CUDA devices (1.869s)
2022-11-23T01:46:33.6197208Z 
2022-11-23T01:46:33.6197405Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6197630Z Ran 1 test in 1.869s
2022-11-23T01:46:33.6197739Z 
2022-11-23T01:46:33.6197812Z OK (skipped=1)
2022-11-23T01:46:33.6197919Z 
2022-11-23T01:46:33.6198003Z Generating XML reports...
2022-11-23T01:46:33.6198407Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014328.xml
2022-11-23T01:46:33.6198864Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6199318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6199647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6200060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6200401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6200563Z 
2022-11-23T01:46:33.6200636Z Running tests...
2022-11-23T01:46:33.6200921Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6201304Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3843
2022-11-23T01:46:33.6201692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3844
2022-11-23T01:46:33.6202141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6202461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6202885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6203229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6203550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6203990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6204310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6204732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6205061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6205381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6205819Z skip: Need at least 2 CUDA devices (1.876s)
2022-11-23T01:46:33.6205956Z 
2022-11-23T01:46:33.6206155Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6206379Z Ran 1 test in 1.876s
2022-11-23T01:46:33.6206490Z 
2022-11-23T01:46:33.6206561Z OK (skipped=1)
2022-11-23T01:46:33.6206665Z 
2022-11-23T01:46:33.6206748Z Generating XML reports...
2022-11-23T01:46:33.6207147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014332.xml
2022-11-23T01:46:33.6207606Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6208102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6208432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6208846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6209193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6209354Z 
2022-11-23T01:46:33.6209426Z Running tests...
2022-11-23T01:46:33.6209709Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6210080Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3876
2022-11-23T01:46:33.6210458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3877
2022-11-23T01:46:33.6210905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6211219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6211646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6211991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6212312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6212756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6213083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6213503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6213832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6214154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6214440Z skip: Need at least 2 CUDA devices (1.767s)
2022-11-23T01:46:33.6214576Z 
2022-11-23T01:46:33.6214770Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6215162Z Ran 1 test in 1.768s
2022-11-23T01:46:33.6215302Z 
2022-11-23T01:46:33.6215377Z OK (skipped=1)
2022-11-23T01:46:33.6215484Z 
2022-11-23T01:46:33.6215567Z Generating XML reports...
2022-11-23T01:46:33.6215975Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014335.xml
2022-11-23T01:46:33.6216436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6216893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6217221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6217633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6217975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6218138Z 
2022-11-23T01:46:33.6218209Z Running tests...
2022-11-23T01:46:33.6218550Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6218934Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6219173Z 
2022-11-23T01:46:33.6219367Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6219602Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6219699Z 
2022-11-23T01:46:33.6219772Z OK (skipped=1)
2022-11-23T01:46:33.6219877Z 
2022-11-23T01:46:33.6219961Z Generating XML reports...
2022-11-23T01:46:33.6220424Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014339.xml
2022-11-23T01:46:33.6220966Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6221420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6221748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6222174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6222504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6222666Z 
2022-11-23T01:46:33.6222738Z Running tests...
2022-11-23T01:46:33.6223036Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6223435Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6223678Z 
2022-11-23T01:46:33.6223872Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6224106Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6224216Z 
2022-11-23T01:46:33.6224287Z OK (skipped=1)
2022-11-23T01:46:33.6224392Z 
2022-11-23T01:46:33.6224464Z Generating XML reports...
2022-11-23T01:46:33.6224930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014340.xml
2022-11-23T01:46:33.6225425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6225882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6226193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6226619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6226968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6227131Z 
2022-11-23T01:46:33.6227190Z Running tests...
2022-11-23T01:46:33.6227487Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6227874Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6228110Z 
2022-11-23T01:46:33.6228302Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6228522Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6228631Z 
2022-11-23T01:46:33.6228702Z OK (skipped=1)
2022-11-23T01:46:33.6228808Z 
2022-11-23T01:46:33.6228890Z Generating XML reports...
2022-11-23T01:46:33.6229338Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014342.xml
2022-11-23T01:46:33.6229827Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6230285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6230610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6231023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6231400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6231565Z 
2022-11-23T01:46:33.6231637Z Running tests...
2022-11-23T01:46:33.6231935Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6232316Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6232563Z 
2022-11-23T01:46:33.6232757Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6233025Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6233134Z 
2022-11-23T01:46:33.6233192Z OK (skipped=1)
2022-11-23T01:46:33.6233298Z 
2022-11-23T01:46:33.6233381Z Generating XML reports...
2022-11-23T01:46:33.6233845Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014343.xml
2022-11-23T01:46:33.6234342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6234785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6235112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6235540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6235880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6236031Z 
2022-11-23T01:46:33.6236102Z Running tests...
2022-11-23T01:46:33.6236398Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6236778Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6237016Z 
2022-11-23T01:46:33.6237193Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6237427Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6237538Z 
2022-11-23T01:46:33.6237609Z OK (skipped=1)
2022-11-23T01:46:33.6237714Z 
2022-11-23T01:46:33.6237796Z Generating XML reports...
2022-11-23T01:46:33.6238244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014344.xml
2022-11-23T01:46:33.6238746Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6239203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6239517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6239947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6240289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6240450Z 
2022-11-23T01:46:33.6240522Z Running tests...
2022-11-23T01:46:33.6240807Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6241186Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6241419Z 
2022-11-23T01:46:33.6241612Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6241832Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6241943Z 
2022-11-23T01:46:33.6242014Z OK (skipped=1)
2022-11-23T01:46:33.6242121Z 
2022-11-23T01:46:33.6242205Z Generating XML reports...
2022-11-23T01:46:33.6242670Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014346.xml
2022-11-23T01:46:33.6243151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6243640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6243972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6244400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6244731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6244891Z 
2022-11-23T01:46:33.6244962Z Running tests...
2022-11-23T01:46:33.6245262Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6245696Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6245965Z 
2022-11-23T01:46:33.6246161Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6246396Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6246506Z 
2022-11-23T01:46:33.6246575Z OK (skipped=1)
2022-11-23T01:46:33.6246722Z 
2022-11-23T01:46:33.6246807Z Generating XML reports...
2022-11-23T01:46:33.6247266Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014347.xml
2022-11-23T01:46:33.6247764Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6248207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6248532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6248959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6249302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6249464Z 
2022-11-23T01:46:33.6249523Z Running tests...
2022-11-23T01:46:33.6249820Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6250206Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6250449Z 
2022-11-23T01:46:33.6250637Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6250855Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6250966Z 
2022-11-23T01:46:33.6251037Z OK (skipped=1)
2022-11-23T01:46:33.6251141Z 
2022-11-23T01:46:33.6251225Z Generating XML reports...
2022-11-23T01:46:33.6251680Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014349.xml
2022-11-23T01:46:33.6252180Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6252637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6252968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6253384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6253723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6253885Z 
2022-11-23T01:46:33.6253956Z Running tests...
2022-11-23T01:46:33.6254239Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6254609Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6254836Z 
2022-11-23T01:46:33.6255193Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6255430Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6255529Z 
2022-11-23T01:46:33.6255602Z OK (skipped=1)
2022-11-23T01:46:33.6255707Z 
2022-11-23T01:46:33.6255790Z Generating XML reports...
2022-11-23T01:46:33.6256248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014350.xml
2022-11-23T01:46:33.6256786Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6257251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6257581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6258008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6258338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6258536Z 
2022-11-23T01:46:33.6258609Z Running tests...
2022-11-23T01:46:33.6258909Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6259249Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6259780Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3999
2022-11-23T01:46:33.6260183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4000
2022-11-23T01:46:33.6260629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6260946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6261371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6261715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6262036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6262483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6262809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6263234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6263563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6263881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6264168Z skip: Need at least 2 CUDA devices (1.769s)
2022-11-23T01:46:33.6264301Z 
2022-11-23T01:46:33.6264495Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6264717Z Ran 1 test in 1.770s
2022-11-23T01:46:33.6264830Z 
2022-11-23T01:46:33.6264904Z OK (skipped=1)
2022-11-23T01:46:33.6265009Z 
2022-11-23T01:46:33.6265093Z Generating XML reports...
2022-11-23T01:46:33.6265543Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014352.xml
2022-11-23T01:46:33.6266037Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6266495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6266822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6267232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6267572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6267733Z 
2022-11-23T01:46:33.6267805Z Running tests...
2022-11-23T01:46:33.6268101Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6268425Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6268856Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4032
2022-11-23T01:46:33.6269286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4033
2022-11-23T01:46:33.6269723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6270049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6270474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6270820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6271128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6271615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6271940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6272352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6272694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6273012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6273300Z skip: Need at least 2 CUDA devices (1.869s)
2022-11-23T01:46:33.6273433Z 
2022-11-23T01:46:33.6273617Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6273855Z Ran 1 test in 1.869s
2022-11-23T01:46:33.6273965Z 
2022-11-23T01:46:33.6274036Z OK (skipped=1)
2022-11-23T01:46:33.6274141Z 
2022-11-23T01:46:33.6274214Z Generating XML reports...
2022-11-23T01:46:33.6274680Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014355.xml
2022-11-23T01:46:33.6275183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6275638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6275951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6276377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6276723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6276886Z 
2022-11-23T01:46:33.6276958Z Running tests...
2022-11-23T01:46:33.6277241Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6277577Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6277997Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4065
2022-11-23T01:46:33.6278369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4066
2022-11-23T01:46:33.6278816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6279148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6279573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6279905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6280227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6280683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6280997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6281419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6281760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6282109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6282386Z skip: Need at least 2 CUDA devices (1.770s)
2022-11-23T01:46:33.6282521Z 
2022-11-23T01:46:33.6282719Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6282959Z Ran 1 test in 1.771s
2022-11-23T01:46:33.6283069Z 
2022-11-23T01:46:33.6283128Z OK (skipped=1)
2022-11-23T01:46:33.6283234Z 
2022-11-23T01:46:33.6283318Z Generating XML reports...
2022-11-23T01:46:33.6283780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014359.xml
2022-11-23T01:46:33.6284307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6284751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6285078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6285501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6285918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6286069Z 
2022-11-23T01:46:33.6286140Z Running tests...
2022-11-23T01:46:33.6286438Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6286773Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6287180Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4098
2022-11-23T01:46:33.6287574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4099
2022-11-23T01:46:33.6288022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6288352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6288769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6289117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6289439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6289883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6290206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6290635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6290977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6291284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6291575Z skip: Need at least 2 CUDA devices (1.769s)
2022-11-23T01:46:33.6291710Z 
2022-11-23T01:46:33.6291907Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6292131Z Ran 1 test in 1.769s
2022-11-23T01:46:33.6292242Z 
2022-11-23T01:46:33.6292315Z OK (skipped=1)
2022-11-23T01:46:33.6292421Z 
2022-11-23T01:46:33.6292504Z Generating XML reports...
2022-11-23T01:46:33.6292967Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014402.xml
2022-11-23T01:46:33.6293451Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6293910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6294242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6294708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6295256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6295419Z 
2022-11-23T01:46:33.6295493Z Running tests...
2022-11-23T01:46:33.6295799Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6296139Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6296674Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4131
2022-11-23T01:46:33.6297133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4132
2022-11-23T01:46:33.6297583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6297901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6298334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6298680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6298990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6299446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6299772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6300198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6300536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6300856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6301146Z skip: Need at least 2 CUDA devices (1.870s)
2022-11-23T01:46:33.6301283Z 
2022-11-23T01:46:33.6301480Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6301705Z Ran 1 test in 1.870s
2022-11-23T01:46:33.6301819Z 
2022-11-23T01:46:33.6301891Z OK (skipped=1)
2022-11-23T01:46:33.6301999Z 
2022-11-23T01:46:33.6302083Z Generating XML reports...
2022-11-23T01:46:33.6302534Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014405.xml
2022-11-23T01:46:33.6303036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6303498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6303825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6304235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6304582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6304744Z 
2022-11-23T01:46:33.6304816Z Running tests...
2022-11-23T01:46:33.6305098Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6305453Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6305989Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4164
2022-11-23T01:46:33.6306387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4165
2022-11-23T01:46:33.6306824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6307150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6307621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6307951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6308271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6308735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6309055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6309465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6309858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6310177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6310465Z skip: Need at least 2 CUDA devices (1.773s)
2022-11-23T01:46:33.6310586Z 
2022-11-23T01:46:33.6310786Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6311026Z Ran 1 test in 1.773s
2022-11-23T01:46:33.6311135Z 
2022-11-23T01:46:33.6311205Z OK (skipped=1)
2022-11-23T01:46:33.6311311Z 
2022-11-23T01:46:33.6311382Z Generating XML reports...
2022-11-23T01:46:33.6311844Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014409.xml
2022-11-23T01:46:33.6312342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6312806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6313123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6313547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6313887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6314049Z 
2022-11-23T01:46:33.6314110Z Running tests...
2022-11-23T01:46:33.6314406Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6314741Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6315282Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4197
2022-11-23T01:46:33.6315679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4198
2022-11-23T01:46:33.6316131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6316458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6316871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6317214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6317540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6317997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6337793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6338395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6338753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6339077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6339370Z skip: Need at least 2 CUDA devices (1.770s)
2022-11-23T01:46:33.6339504Z 
2022-11-23T01:46:33.6339706Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6339934Z Ran 1 test in 1.770s
2022-11-23T01:46:33.6340044Z 
2022-11-23T01:46:33.6340232Z OK (skipped=1)
2022-11-23T01:46:33.6340340Z 
2022-11-23T01:46:33.6340424Z Generating XML reports...
2022-11-23T01:46:33.6340894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014412.xml
2022-11-23T01:46:33.6341385Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6341846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6342177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6342637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6342976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6343139Z 
2022-11-23T01:46:33.6343210Z Running tests...
2022-11-23T01:46:33.6343511Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6343835Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6344377Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4230
2022-11-23T01:46:33.6344783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4231
2022-11-23T01:46:33.6345232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6345553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6345983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6346327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6346638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6347097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6347422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6347845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6348175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6348495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6348784Z skip: Need at least 2 CUDA devices (1.870s)
2022-11-23T01:46:33.6348918Z 
2022-11-23T01:46:33.6349103Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6349337Z Ran 1 test in 1.870s
2022-11-23T01:46:33.6349448Z 
2022-11-23T01:46:33.6349519Z OK (skipped=1)
2022-11-23T01:46:33.6349624Z 
2022-11-23T01:46:33.6349707Z Generating XML reports...
2022-11-23T01:46:33.6350158Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014415.xml
2022-11-23T01:46:33.6350655Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6351111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6351425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6351850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6352195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6352358Z 
2022-11-23T01:46:33.6352428Z Running tests...
2022-11-23T01:46:33.6352712Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6353080Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6353511Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4263
2022-11-23T01:46:33.6353906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4264
2022-11-23T01:46:33.6354344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6354668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6355144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6355473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6355795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6356251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6356572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6356983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6357326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6357643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6357920Z skip: Need at least 2 CUDA devices (1.868s)
2022-11-23T01:46:33.6358055Z 
2022-11-23T01:46:33.6358249Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6358492Z Ran 1 test in 1.868s
2022-11-23T01:46:33.6358601Z 
2022-11-23T01:46:33.6358671Z OK (skipped=1)
2022-11-23T01:46:33.6358765Z 
2022-11-23T01:46:33.6358846Z Generating XML reports...
2022-11-23T01:46:33.6359309Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014419.xml
2022-11-23T01:46:33.6359805Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6360247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6360572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6360996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6361331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6361491Z 
2022-11-23T01:46:33.6361550Z Running tests...
2022-11-23T01:46:33.6361842Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6362182Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6362620Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4296
2022-11-23T01:46:33.6363006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4297
2022-11-23T01:46:33.6363455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6363779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6364193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6364536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6364857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6365312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6365747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6366179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6366521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6366828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6367116Z skip: Need at least 2 CUDA devices (1.867s)
2022-11-23T01:46:33.6367251Z 
2022-11-23T01:46:33.6367445Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6367713Z Ran 1 test in 1.867s
2022-11-23T01:46:33.6367812Z 
2022-11-23T01:46:33.6367885Z OK (skipped=1)
2022-11-23T01:46:33.6367987Z 
2022-11-23T01:46:33.6368070Z Generating XML reports...
2022-11-23T01:46:33.6368533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014422.xml
2022-11-23T01:46:33.6369021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6369477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6369802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6370221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6370548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6370711Z 
2022-11-23T01:46:33.6370781Z Running tests...
2022-11-23T01:46:33.6371078Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6371412Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6371847Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4329
2022-11-23T01:46:33.6372242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4330
2022-11-23T01:46:33.6372687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6373001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6373423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6373763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6374084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6374524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6374992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6375485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6375812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6376130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6376422Z skip: Need at least 2 CUDA devices (1.769s)
2022-11-23T01:46:33.6376556Z 
2022-11-23T01:46:33.6376753Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6376979Z Ran 1 test in 1.770s
2022-11-23T01:46:33.6377091Z 
2022-11-23T01:46:33.6377163Z OK (skipped=1)
2022-11-23T01:46:33.6377268Z 
2022-11-23T01:46:33.6377352Z Generating XML reports...
2022-11-23T01:46:33.6377802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014426.xml
2022-11-23T01:46:33.6378365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6378828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6379153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6379568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6379910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6380073Z 
2022-11-23T01:46:33.6380144Z Running tests...
2022-11-23T01:46:33.6380481Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6380817Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6381232Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4362
2022-11-23T01:46:33.6381611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4363
2022-11-23T01:46:33.6382048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6382372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6382802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6383145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6383453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6383914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6384237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6384648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6384986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6385306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6385591Z skip: Need at least 2 CUDA devices (1.773s)
2022-11-23T01:46:33.6385728Z 
2022-11-23T01:46:33.6385913Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6386149Z Ran 1 test in 1.773s
2022-11-23T01:46:33.6386258Z 
2022-11-23T01:46:33.6386327Z OK (skipped=1)
2022-11-23T01:46:33.6386434Z 
2022-11-23T01:46:33.6386504Z Generating XML reports...
2022-11-23T01:46:33.6386963Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014429.xml
2022-11-23T01:46:33.6387460Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6387917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6388229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6388652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6388994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6389155Z 
2022-11-23T01:46:33.6389225Z Running tests...
2022-11-23T01:46:33.6389509Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6389857Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6390269Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4395
2022-11-23T01:46:33.6390634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4396
2022-11-23T01:46:33.6391110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6391436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6391864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6392193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6392513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6392966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6393307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6393730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6394071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6394391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6394664Z skip: Need at least 2 CUDA devices (1.872s)
2022-11-23T01:46:33.6394796Z 
2022-11-23T01:46:33.6394990Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6395225Z Ran 1 test in 1.872s
2022-11-23T01:46:33.6395333Z 
2022-11-23T01:46:33.6395392Z OK (skipped=1)
2022-11-23T01:46:33.6395496Z 
2022-11-23T01:46:33.6395578Z Generating XML reports...
2022-11-23T01:46:33.6396038Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014432.xml
2022-11-23T01:46:33.6396537Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6397001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6397324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6397749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6398089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6398238Z 
2022-11-23T01:46:33.6398308Z Running tests...
2022-11-23T01:46:33.6398601Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6398979Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6399216Z 
2022-11-23T01:46:33.6399395Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6399466Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6399471Z 
2022-11-23T01:46:33.6399539Z OK (skipped=1)
2022-11-23T01:46:33.6399544Z 
2022-11-23T01:46:33.6399626Z Generating XML reports...
2022-11-23T01:46:33.6399976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014436.xml
2022-11-23T01:46:33.6400203Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6400478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6400598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6400880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6401005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6401010Z 
2022-11-23T01:46:33.6401081Z Running tests...
2022-11-23T01:46:33.6401271Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6401538Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6401544Z 
2022-11-23T01:46:33.6401735Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6401808Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6401813Z 
2022-11-23T01:46:33.6401881Z OK (skipped=1)
2022-11-23T01:46:33.6401886Z 
2022-11-23T01:46:33.6401968Z Generating XML reports...
2022-11-23T01:46:33.6402300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014437.xml
2022-11-23T01:46:33.6402529Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6402854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6402978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6403258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6403391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6403396Z 
2022-11-23T01:46:33.6403465Z Running tests...
2022-11-23T01:46:33.6403658Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6403891Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6403896Z 
2022-11-23T01:46:33.6404086Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6404149Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6404154Z 
2022-11-23T01:46:33.6404223Z OK (skipped=1)
2022-11-23T01:46:33.6404227Z 
2022-11-23T01:46:33.6404311Z Generating XML reports...
2022-11-23T01:46:33.6404653Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014439.xml
2022-11-23T01:46:33.6404886Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6405163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6405287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6405653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6405778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6405798Z 
2022-11-23T01:46:33.6405856Z Running tests...
2022-11-23T01:46:33.6406049Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6406218Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6406441Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6406447Z 
2022-11-23T01:46:33.6406637Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6406711Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6406716Z 
2022-11-23T01:46:33.6406784Z OK (skipped=1)
2022-11-23T01:46:33.6406789Z 
2022-11-23T01:46:33.6406870Z Generating XML reports...
2022-11-23T01:46:33.6407200Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014440.xml
2022-11-23T01:46:33.6407431Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6407707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6407832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6408112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6408280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6408285Z 
2022-11-23T01:46:33.6408358Z Running tests...
2022-11-23T01:46:33.6408551Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6408715Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6408925Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6408931Z 
2022-11-23T01:46:33.6409148Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6409221Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6409226Z 
2022-11-23T01:46:33.6409296Z OK (skipped=1)
2022-11-23T01:46:33.6409300Z 
2022-11-23T01:46:33.6409383Z Generating XML reports...
2022-11-23T01:46:33.6409730Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014442.xml
2022-11-23T01:46:33.6409958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6410228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6410347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6410614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6410747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6410754Z 
2022-11-23T01:46:33.6410828Z Running tests...
2022-11-23T01:46:33.6411019Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6411239Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6411245Z 
2022-11-23T01:46:33.6411437Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6411512Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6411516Z 
2022-11-23T01:46:33.6411585Z OK (skipped=1)
2022-11-23T01:46:33.6411590Z 
2022-11-23T01:46:33.6411660Z Generating XML reports...
2022-11-23T01:46:33.6412000Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014443.xml
2022-11-23T01:46:33.6412226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6412499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6412619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6412897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6413030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6413035Z 
2022-11-23T01:46:33.6413105Z Running tests...
2022-11-23T01:46:33.6413295Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6413428Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6413622Z Tests that DDP with ``device_ids`` specified can run a forward and ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6413627Z 
2022-11-23T01:46:33.6413816Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6413893Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6413898Z 
2022-11-23T01:46:33.6413966Z OK (skipped=1)
2022-11-23T01:46:33.6413971Z 
2022-11-23T01:46:33.6414052Z Generating XML reports...
2022-11-23T01:46:33.6414393Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014444.xml
2022-11-23T01:46:33.6414647Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6415063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6415198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6415483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6415617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6415623Z 
2022-11-23T01:46:33.6415747Z Running tests...
2022-11-23T01:46:33.6415940Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6416151Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6416156Z 
2022-11-23T01:46:33.6416345Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6416422Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6416428Z 
2022-11-23T01:46:33.6416498Z OK (skipped=1)
2022-11-23T01:46:33.6416502Z 
2022-11-23T01:46:33.6416572Z Generating XML reports...
2022-11-23T01:46:33.6416920Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014446.xml
2022-11-23T01:46:33.6417151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6417426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6417550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6417833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6417967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6417972Z 
2022-11-23T01:46:33.6418046Z Running tests...
2022-11-23T01:46:33.6418228Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6418438Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6418443Z 
2022-11-23T01:46:33.6418631Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6418706Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6418712Z 
2022-11-23T01:46:33.6418780Z OK (skipped=1)
2022-11-23T01:46:33.6418784Z 
2022-11-23T01:46:33.6418864Z Generating XML reports...
2022-11-23T01:46:33.6419209Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014447.xml
2022-11-23T01:46:33.6419436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6419709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6419818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6420098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6420231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6420235Z 
2022-11-23T01:46:33.6420307Z Running tests...
2022-11-23T01:46:33.6420497Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6420710Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6420717Z 
2022-11-23T01:46:33.6420902Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6420973Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6420978Z 
2022-11-23T01:46:33.6421047Z OK (skipped=1)
2022-11-23T01:46:33.6421051Z 
2022-11-23T01:46:33.6421160Z Generating XML reports...
2022-11-23T01:46:33.6421500Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014449.xml
2022-11-23T01:46:33.6421725Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6421998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6422120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6422400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6422563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6422568Z 
2022-11-23T01:46:33.6422637Z Running tests...
2022-11-23T01:46:33.6422828Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6423039Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6423044Z 
2022-11-23T01:46:33.6423233Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6423304Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6423309Z 
2022-11-23T01:46:33.6423378Z OK (skipped=1)
2022-11-23T01:46:33.6423383Z 
2022-11-23T01:46:33.6423465Z Generating XML reports...
2022-11-23T01:46:33.6423808Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014450.xml
2022-11-23T01:46:33.6424041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6424312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6424434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6424708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6424842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6424846Z 
2022-11-23T01:46:33.6424916Z Running tests...
2022-11-23T01:46:33.6425106Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6425308Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6425314Z 
2022-11-23T01:46:33.6425501Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6425576Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6425581Z 
2022-11-23T01:46:33.6425650Z OK (skipped=1)
2022-11-23T01:46:33.6425654Z 
2022-11-23T01:46:33.6425723Z Generating XML reports...
2022-11-23T01:46:33.6426068Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014452.xml
2022-11-23T01:46:33.6426297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6426568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6426689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6426967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6427101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6427108Z 
2022-11-23T01:46:33.6427179Z Running tests...
2022-11-23T01:46:33.6427368Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6427587Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6427604Z 
2022-11-23T01:46:33.6427807Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6427881Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6427886Z 
2022-11-23T01:46:33.6427957Z OK (skipped=1)
2022-11-23T01:46:33.6427961Z 
2022-11-23T01:46:33.6428043Z Generating XML reports...
2022-11-23T01:46:33.6428385Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014453.xml
2022-11-23T01:46:33.6428612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6428880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6429028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6429295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6429428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6429435Z 
2022-11-23T01:46:33.6429506Z Running tests...
2022-11-23T01:46:33.6429699Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6429926Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6429931Z 
2022-11-23T01:46:33.6430120Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6430193Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6430197Z 
2022-11-23T01:46:33.6430266Z OK (skipped=1)
2022-11-23T01:46:33.6430272Z 
2022-11-23T01:46:33.6430355Z Generating XML reports...
2022-11-23T01:46:33.6430687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014455.xml
2022-11-23T01:46:33.6430917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6431193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6431314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6431590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6431724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6431729Z 
2022-11-23T01:46:33.6431799Z Running tests...
2022-11-23T01:46:33.6431989Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6432207Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6432224Z 
2022-11-23T01:46:33.6432401Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6432473Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6432477Z 
2022-11-23T01:46:33.6432548Z OK (skipped=1)
2022-11-23T01:46:33.6432554Z 
2022-11-23T01:46:33.6432637Z Generating XML reports...
2022-11-23T01:46:33.6432982Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014456.xml
2022-11-23T01:46:33.6433210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6433478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6433601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6433871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6434006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6434011Z 
2022-11-23T01:46:33.6434081Z Running tests...
2022-11-23T01:46:33.6434300Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6434548Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6434553Z 
2022-11-23T01:46:33.6434742Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6434816Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6434821Z 
2022-11-23T01:46:33.6434892Z OK (skipped=1)
2022-11-23T01:46:33.6434896Z 
2022-11-23T01:46:33.6434979Z Generating XML reports...
2022-11-23T01:46:33.6435309Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014457.xml
2022-11-23T01:46:33.6435564Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6435835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6435960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6436238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6436371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6436376Z 
2022-11-23T01:46:33.6436446Z Running tests...
2022-11-23T01:46:33.6436636Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6436874Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6436881Z 
2022-11-23T01:46:33.6437060Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6437134Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6437139Z 
2022-11-23T01:46:33.6437213Z OK (skipped=1)
2022-11-23T01:46:33.6437217Z 
2022-11-23T01:46:33.6437301Z Generating XML reports...
2022-11-23T01:46:33.6437645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014459.xml
2022-11-23T01:46:33.6437872Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6438145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6438266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6438543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6438667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6438681Z 
2022-11-23T01:46:33.6438742Z Running tests...
2022-11-23T01:46:33.6438934Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6439171Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6439177Z 
2022-11-23T01:46:33.6439365Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6439437Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6439441Z 
2022-11-23T01:46:33.6439510Z OK (skipped=1)
2022-11-23T01:46:33.6439514Z 
2022-11-23T01:46:33.6439594Z Generating XML reports...
2022-11-23T01:46:33.6439937Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014500.xml
2022-11-23T01:46:33.6440152Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6440422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6440542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6440857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6440991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6440995Z 
2022-11-23T01:46:33.6441065Z Running tests...
2022-11-23T01:46:33.6441257Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6441445Z   test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6441450Z 
2022-11-23T01:46:33.6441639Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6441731Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6441736Z 
2022-11-23T01:46:33.6441805Z OK (skipped=1)
2022-11-23T01:46:33.6441809Z 
2022-11-23T01:46:33.6441890Z Generating XML reports...
2022-11-23T01:46:33.6442234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014502.xml
2022-11-23T01:46:33.6442463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6442734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6442855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6443138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6443259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6443275Z 
2022-11-23T01:46:33.6443334Z Running tests...
2022-11-23T01:46:33.6443528Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6443747Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6443753Z 
2022-11-23T01:46:33.6443944Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6444020Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6444025Z 
2022-11-23T01:46:33.6444094Z OK (skipped=1)
2022-11-23T01:46:33.6444098Z 
2022-11-23T01:46:33.6444180Z Generating XML reports...
2022-11-23T01:46:33.6444524Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014503.xml
2022-11-23T01:46:33.6444741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6445010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6445133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6445413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6445619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6445625Z 
2022-11-23T01:46:33.6445700Z Running tests...
2022-11-23T01:46:33.6445894Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6446108Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6446113Z 
2022-11-23T01:46:33.6446305Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6446365Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6446370Z 
2022-11-23T01:46:33.6446439Z OK (skipped=1)
2022-11-23T01:46:33.6446443Z 
2022-11-23T01:46:33.6446525Z Generating XML reports...
2022-11-23T01:46:33.6446873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014505.xml
2022-11-23T01:46:33.6447100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6447406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6447530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6447805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6447936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6447941Z 
2022-11-23T01:46:33.6447999Z Running tests...
2022-11-23T01:46:33.6448190Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6448390Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6448440Z 
2022-11-23T01:46:33.6448633Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6448706Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6448711Z 
2022-11-23T01:46:33.6448781Z OK (skipped=1)
2022-11-23T01:46:33.6448785Z 
2022-11-23T01:46:33.6448868Z Generating XML reports...
2022-11-23T01:46:33.6449214Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014506.xml
2022-11-23T01:46:33.6449442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6449704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6449825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6450105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6450242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6450247Z 
2022-11-23T01:46:33.6450317Z Running tests...
2022-11-23T01:46:33.6450507Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6450747Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6450752Z 
2022-11-23T01:46:33.6450943Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6451004Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6451021Z 
2022-11-23T01:46:33.6451077Z OK (skipped=1)
2022-11-23T01:46:33.6451081Z 
2022-11-23T01:46:33.6451163Z Generating XML reports...
2022-11-23T01:46:33.6451506Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014508.xml
2022-11-23T01:46:33.6451735Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6452007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6452128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6452406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6452537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6452542Z 
2022-11-23T01:46:33.6452601Z Running tests...
2022-11-23T01:46:33.6452791Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6453004Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6453009Z 
2022-11-23T01:46:33.6453197Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6453276Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6453280Z 
2022-11-23T01:46:33.6453350Z OK (skipped=1)
2022-11-23T01:46:33.6453354Z 
2022-11-23T01:46:33.6453436Z Generating XML reports...
2022-11-23T01:46:33.6453773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014509.xml
2022-11-23T01:46:33.6454030Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6454289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6454409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6454686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6454817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6454966Z 
2022-11-23T01:46:33.6455080Z Running tests...
2022-11-23T01:46:33.6455278Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6455520Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4668
2022-11-23T01:46:33.6455677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4669
2022-11-23T01:46:33.6455950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6456060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6456341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6456473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6456641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6457055Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6457464Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6457871Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6458279Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6458684Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6459084Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6459362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6459537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6459820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6459956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6460109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6460515Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6461003Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6461408Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6461804Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6462216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6462614Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:46:33.6462685Z ok (1.766s)
2022-11-23T01:46:33.6462693Z 
2022-11-23T01:46:33.6462896Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6462971Z Ran 1 test in 1.766s
2022-11-23T01:46:33.6462977Z 
2022-11-23T01:46:33.6463039Z OK
2022-11-23T01:46:33.6463043Z 
2022-11-23T01:46:33.6463125Z Generating XML reports...
2022-11-23T01:46:33.6463474Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014510.xml
2022-11-23T01:46:33.6463706Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6463983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6464094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6464376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6464511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6464519Z 
2022-11-23T01:46:33.6464591Z Running tests...
2022-11-23T01:46:33.6464784Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6465010Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6465046Z 
2022-11-23T01:46:33.6465241Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6465317Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6465322Z 
2022-11-23T01:46:33.6465391Z OK (skipped=1)
2022-11-23T01:46:33.6465395Z 
2022-11-23T01:46:33.6465467Z Generating XML reports...
2022-11-23T01:46:33.6465817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014514.xml
2022-11-23T01:46:33.6466047Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6466350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6466476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6466757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6466890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6466896Z 
2022-11-23T01:46:33.6466966Z Running tests...
2022-11-23T01:46:33.6467160Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6467384Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6467388Z 
2022-11-23T01:46:33.6467584Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6467658Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6467665Z 
2022-11-23T01:46:33.6467734Z OK (skipped=1)
2022-11-23T01:46:33.6467739Z 
2022-11-23T01:46:33.6467820Z Generating XML reports...
2022-11-23T01:46:33.6468163Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014515.xml
2022-11-23T01:46:33.6468392Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6468665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6468785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6469054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6469187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6469193Z 
2022-11-23T01:46:33.6469263Z Running tests...
2022-11-23T01:46:33.6469456Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6469691Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6469696Z 
2022-11-23T01:46:33.6469886Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6469960Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6469965Z 
2022-11-23T01:46:33.6470033Z OK (skipped=1)
2022-11-23T01:46:33.6470038Z 
2022-11-23T01:46:33.6470119Z Generating XML reports...
2022-11-23T01:46:33.6470454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014517.xml
2022-11-23T01:46:33.6470682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6471009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6471133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6471412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6471544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6471548Z 
2022-11-23T01:46:33.6471618Z Running tests...
2022-11-23T01:46:33.6471838Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6472065Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6472082Z 
2022-11-23T01:46:33.6472260Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6472333Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6472338Z 
2022-11-23T01:46:33.6472406Z OK (skipped=1)
2022-11-23T01:46:33.6472411Z 
2022-11-23T01:46:33.6472495Z Generating XML reports...
2022-11-23T01:46:33.6472865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014518.xml
2022-11-23T01:46:33.6473092Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6473366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6473489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6473757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6473890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6473895Z 
2022-11-23T01:46:33.6473965Z Running tests...
2022-11-23T01:46:33.6474157Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6474367Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6474374Z 
2022-11-23T01:46:33.6474561Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6474634Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6474638Z 
2022-11-23T01:46:33.6474706Z OK (skipped=1)
2022-11-23T01:46:33.6474711Z 
2022-11-23T01:46:33.6474795Z Generating XML reports...
2022-11-23T01:46:33.6475130Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014520.xml
2022-11-23T01:46:33.6475358Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6475628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6475749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6476027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6476164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6476169Z 
2022-11-23T01:46:33.6476238Z Running tests...
2022-11-23T01:46:33.6476429Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6476642Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6476647Z 
2022-11-23T01:46:33.6476824Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6476895Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6476900Z 
2022-11-23T01:46:33.6476969Z OK (skipped=1)
2022-11-23T01:46:33.6476973Z 
2022-11-23T01:46:33.6477058Z Generating XML reports...
2022-11-23T01:46:33.6477398Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014521.xml
2022-11-23T01:46:33.6477627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6477899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6478020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6478315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6478451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6478455Z 
2022-11-23T01:46:33.6478525Z Running tests...
2022-11-23T01:46:33.6478715Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6478943Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6478948Z 
2022-11-23T01:46:33.6479135Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6479234Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6479239Z 
2022-11-23T01:46:33.6479307Z OK (skipped=1)
2022-11-23T01:46:33.6479312Z 
2022-11-23T01:46:33.6479392Z Generating XML reports...
2022-11-23T01:46:33.6479720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014522.xml
2022-11-23T01:46:33.6479949Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6480218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6480341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6480617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6480751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6480758Z 
2022-11-23T01:46:33.6480828Z Running tests...
2022-11-23T01:46:33.6481020Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6481255Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6481261Z 
2022-11-23T01:46:33.6481452Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6481514Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6481519Z 
2022-11-23T01:46:33.6481586Z OK (skipped=1)
2022-11-23T01:46:33.6481591Z 
2022-11-23T01:46:33.6481673Z Generating XML reports...
2022-11-23T01:46:33.6482012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014524.xml
2022-11-23T01:46:33.6482238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6482509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6482634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6482911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6483033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6483048Z 
2022-11-23T01:46:33.6483107Z Running tests...
2022-11-23T01:46:33.6483295Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6483533Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6483538Z 
2022-11-23T01:46:33.6483729Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6483800Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6483805Z 
2022-11-23T01:46:33.6483872Z OK (skipped=1)
2022-11-23T01:46:33.6483877Z 
2022-11-23T01:46:33.6483957Z Generating XML reports...
2022-11-23T01:46:33.6484300Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014525.xml
2022-11-23T01:46:33.6484517Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6484816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6484938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6485219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6485351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6485356Z 
2022-11-23T01:46:33.6485426Z Running tests...
2022-11-23T01:46:33.6485692Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6485970Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6485975Z 
2022-11-23T01:46:33.6486171Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6486231Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6486236Z 
2022-11-23T01:46:33.6486305Z OK (skipped=1)
2022-11-23T01:46:33.6486310Z 
2022-11-23T01:46:33.6486392Z Generating XML reports...
2022-11-23T01:46:33.6486735Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014527.xml
2022-11-23T01:46:33.6486962Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6487235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6487352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6487632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6487767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6487772Z 
2022-11-23T01:46:33.6487831Z Running tests...
2022-11-23T01:46:33.6488024Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6488242Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6488247Z 
2022-11-23T01:46:33.6488443Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6488518Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6488523Z 
2022-11-23T01:46:33.6488592Z OK (skipped=1)
2022-11-23T01:46:33.6488596Z 
2022-11-23T01:46:33.6488679Z Generating XML reports...
2022-11-23T01:46:33.6489022Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014528.xml
2022-11-23T01:46:33.6489245Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6489516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6489639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6489917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6490052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6490057Z 
2022-11-23T01:46:33.6490127Z Running tests...
2022-11-23T01:46:33.6490319Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6490447Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-11-23T01:46:33.6490639Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6490646Z 
2022-11-23T01:46:33.6490826Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6490899Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6490904Z 
2022-11-23T01:46:33.6490972Z OK (skipped=1)
2022-11-23T01:46:33.6490976Z 
2022-11-23T01:46:33.6491089Z Generating XML reports...
2022-11-23T01:46:33.6491437Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014530.xml
2022-11-23T01:46:33.6491665Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6491938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6492059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6492338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6492502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6492506Z 
2022-11-23T01:46:33.6492577Z Running tests...
2022-11-23T01:46:33.6492767Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6492985Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6492991Z 
2022-11-23T01:46:33.6493182Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6493252Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6493257Z 
2022-11-23T01:46:33.6493326Z OK (skipped=1)
2022-11-23T01:46:33.6493331Z 
2022-11-23T01:46:33.6493411Z Generating XML reports...
2022-11-23T01:46:33.6493751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014531.xml
2022-11-23T01:46:33.6493968Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6494235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6494352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6494634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6494761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6494766Z 
2022-11-23T01:46:33.6494836Z Running tests...
2022-11-23T01:46:33.6495127Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6495331Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6495336Z 
2022-11-23T01:46:33.6495524Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6495587Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6495592Z 
2022-11-23T01:46:33.6495655Z OK (skipped=1)
2022-11-23T01:46:33.6495660Z 
2022-11-23T01:46:33.6495744Z Generating XML reports...
2022-11-23T01:46:33.6496086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014533.xml
2022-11-23T01:46:33.6496313Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6496586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6496703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6496983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6497103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6497120Z 
2022-11-23T01:46:33.6497178Z Running tests...
2022-11-23T01:46:33.6497366Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6497583Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6497588Z 
2022-11-23T01:46:33.6497823Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6497899Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6497904Z 
2022-11-23T01:46:33.6497968Z OK (skipped=1)
2022-11-23T01:46:33.6497973Z 
2022-11-23T01:46:33.6498051Z Generating XML reports...
2022-11-23T01:46:33.6498390Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014534.xml
2022-11-23T01:46:33.6498607Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6498878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6499030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6499308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6499439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6499446Z 
2022-11-23T01:46:33.6499515Z Running tests...
2022-11-23T01:46:33.6499702Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6499926Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6499931Z 
2022-11-23T01:46:33.6500116Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6500177Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6500181Z 
2022-11-23T01:46:33.6500245Z OK (skipped=1)
2022-11-23T01:46:33.6500252Z 
2022-11-23T01:46:33.6500333Z Generating XML reports...
2022-11-23T01:46:33.6500677Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014535.xml
2022-11-23T01:46:33.6500901Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6501174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6501293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6501573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6501695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6501707Z 
2022-11-23T01:46:33.6501766Z Running tests...
2022-11-23T01:46:33.6501955Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6502202Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4861
2022-11-23T01:46:33.6502354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4862
2022-11-23T01:46:33.6502625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6502744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6503018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6503151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6503302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6503565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6503688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6503962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6504093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6504285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6504388Z skip: Need at least 2 CUDA devices (1.767s)
2022-11-23T01:46:33.6504393Z 
2022-11-23T01:46:33.6504582Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6504654Z Ran 1 test in 1.767s
2022-11-23T01:46:33.6504659Z 
2022-11-23T01:46:33.6504716Z OK (skipped=1)
2022-11-23T01:46:33.6504720Z 
2022-11-23T01:46:33.6504799Z Generating XML reports...
2022-11-23T01:46:33.6505142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014537.xml
2022-11-23T01:46:33.6505398Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6505670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6505788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6506065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6506196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6506201Z 
2022-11-23T01:46:33.6506261Z Running tests...
2022-11-23T01:46:33.6506451Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6506694Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4894
2022-11-23T01:46:33.6506849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4895
2022-11-23T01:46:33.6507120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6507238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6507518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6507652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6507813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:46:33.6508068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6508188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6508461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6508594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6508751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:46:33.6508852Z skip: Need at least 2 CUDA devices (1.771s)
2022-11-23T01:46:33.6508858Z 
2022-11-23T01:46:33.6509050Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6509118Z Ran 1 test in 1.771s
2022-11-23T01:46:33.6509123Z 
2022-11-23T01:46:33.6509190Z OK (skipped=1)
2022-11-23T01:46:33.6509195Z 
2022-11-23T01:46:33.6509266Z Generating XML reports...
2022-11-23T01:46:33.6509602Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014540.xml
2022-11-23T01:46:33.6509828Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6510095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6510216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6510497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6510628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6510665Z 
2022-11-23T01:46:33.6510735Z Running tests...
2022-11-23T01:46:33.6510914Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6511118Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6511123Z 
2022-11-23T01:46:33.6511308Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6511382Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6511387Z 
2022-11-23T01:46:33.6511451Z OK (skipped=1)
2022-11-23T01:46:33.6511483Z 
2022-11-23T01:46:33.6511564Z Generating XML reports...
2022-11-23T01:46:33.6511894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014544.xml
2022-11-23T01:46:33.6512121Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6512395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6512506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6512783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6512915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6512920Z 
2022-11-23T01:46:33.6512989Z Running tests...
2022-11-23T01:46:33.6513178Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6513381Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6513386Z 
2022-11-23T01:46:33.6513575Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6513645Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6513650Z 
2022-11-23T01:46:33.6513716Z OK (skipped=1)
2022-11-23T01:46:33.6513721Z 
2022-11-23T01:46:33.6513793Z Generating XML reports...
2022-11-23T01:46:33.6514112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014545.xml
2022-11-23T01:46:33.6514339Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6514610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6514729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6515006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6515137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6515142Z 
2022-11-23T01:46:33.6515212Z Running tests...
2022-11-23T01:46:33.6515390Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6515590Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6515595Z 
2022-11-23T01:46:33.6515780Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6515853Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6515858Z 
2022-11-23T01:46:33.6515929Z OK (skipped=1)
2022-11-23T01:46:33.6515934Z 
2022-11-23T01:46:33.6516015Z Generating XML reports...
2022-11-23T01:46:33.6516338Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014546.xml
2022-11-23T01:46:33.6516568Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6516837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6516947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6517254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6517389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6517394Z 
2022-11-23T01:46:33.6517466Z Running tests...
2022-11-23T01:46:33.6517657Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6517859Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6517864Z 
2022-11-23T01:46:33.6518057Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6518157Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6518162Z 
2022-11-23T01:46:33.6518230Z OK (skipped=1)
2022-11-23T01:46:33.6518235Z 
2022-11-23T01:46:33.6518305Z Generating XML reports...
2022-11-23T01:46:33.6518627Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014548.xml
2022-11-23T01:46:33.6518856Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6519129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6519249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6519526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6519658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6519664Z 
2022-11-23T01:46:33.6519735Z Running tests...
2022-11-23T01:46:33.6519927Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6520122Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6520127Z 
2022-11-23T01:46:33.6520318Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6520391Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6520396Z 
2022-11-23T01:46:33.6520465Z OK (skipped=1)
2022-11-23T01:46:33.6520469Z 
2022-11-23T01:46:33.6520550Z Generating XML reports...
2022-11-23T01:46:33.6520873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014549.xml
2022-11-23T01:46:33.6521100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6521372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6521496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6521763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6521896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6521903Z 
2022-11-23T01:46:33.6521976Z Running tests...
2022-11-23T01:46:33.6522168Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6522368Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6522373Z 
2022-11-23T01:46:33.6522562Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6522634Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6522639Z 
2022-11-23T01:46:33.6522706Z OK (skipped=1)
2022-11-23T01:46:33.6522713Z 
2022-11-23T01:46:33.6522783Z Generating XML reports...
2022-11-23T01:46:33.6523102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014551.xml
2022-11-23T01:46:33.6523327Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6523626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6523752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6524029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6524161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6524166Z 
2022-11-23T01:46:33.6524235Z Running tests...
2022-11-23T01:46:33.6524424Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6524643Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6524659Z 
2022-11-23T01:46:33.6524837Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6524909Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6524914Z 
2022-11-23T01:46:33.6524984Z OK (skipped=1)
2022-11-23T01:46:33.6524989Z 
2022-11-23T01:46:33.6525073Z Generating XML reports...
2022-11-23T01:46:33.6525397Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014552.xml
2022-11-23T01:46:33.6525697Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6525979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6526100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6526368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6526503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6526507Z 
2022-11-23T01:46:33.6526578Z Running tests...
2022-11-23T01:46:33.6526770Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6526970Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6526975Z 
2022-11-23T01:46:33.6527162Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6527236Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6527241Z 
2022-11-23T01:46:33.6527310Z OK (skipped=1)
2022-11-23T01:46:33.6527315Z 
2022-11-23T01:46:33.6527395Z Generating XML reports...
2022-11-23T01:46:33.6527710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014554.xml
2022-11-23T01:46:33.6527940Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6528213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6528334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6528617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6528749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6528754Z 
2022-11-23T01:46:33.6528824Z Running tests...
2022-11-23T01:46:33.6529018Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6529193Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6529209Z 
2022-11-23T01:46:33.6529386Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6529462Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6529467Z 
2022-11-23T01:46:33.6529535Z OK (skipped=1)
2022-11-23T01:46:33.6529540Z 
2022-11-23T01:46:33.6529621Z Generating XML reports...
2022-11-23T01:46:33.6529943Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014555.xml
2022-11-23T01:46:33.6530206Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6530477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6530599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6530866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6530999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6531030Z 
2022-11-23T01:46:33.6531101Z Running tests...
2022-11-23T01:46:33.6531293Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6531547Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6531552Z 
2022-11-23T01:46:33.6531743Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6531816Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6531820Z 
2022-11-23T01:46:33.6531888Z OK (skipped=1)
2022-11-23T01:46:33.6531893Z 
2022-11-23T01:46:33.6531974Z Generating XML reports...
2022-11-23T01:46:33.6532379Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014557.xml
2022-11-23T01:46:33.6532607Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6532879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6533003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6533283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6533416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6533424Z 
2022-11-23T01:46:33.6533494Z Running tests...
2022-11-23T01:46:33.6533689Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6533949Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6533955Z 
2022-11-23T01:46:33.6534133Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6534206Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6534211Z 
2022-11-23T01:46:33.6534280Z OK (skipped=1)
2022-11-23T01:46:33.6534288Z 
2022-11-23T01:46:33.6534370Z Generating XML reports...
2022-11-23T01:46:33.6534786Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014558.xml
2022-11-23T01:46:33.6535127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6535402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6535525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6535803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6535923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6535929Z 
2022-11-23T01:46:33.6535999Z Running tests...
2022-11-23T01:46:33.6536190Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6536443Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6536448Z 
2022-11-23T01:46:33.6536639Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6536712Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6536717Z 
2022-11-23T01:46:33.6536831Z OK (skipped=1)
2022-11-23T01:46:33.6536837Z 
2022-11-23T01:46:33.6536923Z Generating XML reports...
2022-11-23T01:46:33.6537334Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014559.xml
2022-11-23T01:46:33.6537563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6537834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6537955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6538281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6538415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6538420Z 
2022-11-23T01:46:33.6538492Z Running tests...
2022-11-23T01:46:33.6538685Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6538945Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6538951Z 
2022-11-23T01:46:33.6539141Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6539203Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6539208Z 
2022-11-23T01:46:33.6539277Z OK (skipped=1)
2022-11-23T01:46:33.6539282Z 
2022-11-23T01:46:33.6539365Z Generating XML reports...
2022-11-23T01:46:33.6539788Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014601.xml
2022-11-23T01:46:33.6540017Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6540290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6540414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6540694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6540815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6540831Z 
2022-11-23T01:46:33.6540890Z Running tests...
2022-11-23T01:46:33.6541082Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6541278Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6541286Z 
2022-11-23T01:46:33.6541477Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6541549Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6541554Z 
2022-11-23T01:46:33.6541622Z OK (skipped=1)
2022-11-23T01:46:33.6541627Z 
2022-11-23T01:46:33.6541708Z Generating XML reports...
2022-11-23T01:46:33.6542047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123014602.xml
2022-11-23T01:46:33.6542261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6542531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6542652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6542928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6543066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6543071Z 
2022-11-23T01:46:33.6543140Z Running tests...
2022-11-23T01:46:33.6543331Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6543559Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6543565Z 
2022-11-23T01:46:33.6543757Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6543818Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6543823Z 
2022-11-23T01:46:33.6543891Z OK (skipped=1)
2022-11-23T01:46:33.6543895Z 
2022-11-23T01:46:33.6543976Z Generating XML reports...
2022-11-23T01:46:33.6544297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014604.xml
2022-11-23T01:46:33.6544524Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6544823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6544945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6545220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6545343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6545360Z 
2022-11-23T01:46:33.6545419Z Running tests...
2022-11-23T01:46:33.6545611Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6545805Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6545809Z 
2022-11-23T01:46:33.6545999Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6546072Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6546080Z 
2022-11-23T01:46:33.6546149Z OK (skipped=1)
2022-11-23T01:46:33.6546154Z 
2022-11-23T01:46:33.6546234Z Generating XML reports...
2022-11-23T01:46:33.6546554Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014605.xml
2022-11-23T01:46:33.6546770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6547044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6547167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6547446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6547579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6547584Z 
2022-11-23T01:46:33.6547654Z Running tests...
2022-11-23T01:46:33.6547844Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6548032Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6548037Z 
2022-11-23T01:46:33.6548226Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6548288Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6548293Z 
2022-11-23T01:46:33.6548363Z OK (skipped=1)
2022-11-23T01:46:33.6548367Z 
2022-11-23T01:46:33.6548451Z Generating XML reports...
2022-11-23T01:46:33.6548772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014607.xml
2022-11-23T01:46:33.6548999Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6549274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6549394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6549671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6549805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6549810Z 
2022-11-23T01:46:33.6549869Z Running tests...
2022-11-23T01:46:33.6550090Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6550285Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-11-23T01:46:33.6550290Z 
2022-11-23T01:46:33.6550481Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6550555Z Ran 1 test in 0.003s
2022-11-23T01:46:33.6550560Z 
2022-11-23T01:46:33.6550629Z OK (skipped=1)
2022-11-23T01:46:33.6550634Z 
2022-11-23T01:46:33.6550714Z Generating XML reports...
2022-11-23T01:46:33.6551035Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014608.xml
2022-11-23T01:46:33.6551280Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6551551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6551673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6551952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6552085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6552090Z 
2022-11-23T01:46:33.6552160Z Running tests...
2022-11-23T01:46:33.6552350Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6552529Z   test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6552534Z 
2022-11-23T01:46:33.6552724Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6552784Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6552789Z 
2022-11-23T01:46:33.6552858Z OK (skipped=1)
2022-11-23T01:46:33.6552863Z 
2022-11-23T01:46:33.6552943Z Generating XML reports...
2022-11-23T01:46:33.6553264Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014610.xml
2022-11-23T01:46:33.6553493Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6553763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6553884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6554159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6554291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6554300Z 
2022-11-23T01:46:33.6554358Z Running tests...
2022-11-23T01:46:33.6554549Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6554735Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6554739Z 
2022-11-23T01:46:33.6554932Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6555005Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6555010Z 
2022-11-23T01:46:33.6555081Z OK (skipped=1)
2022-11-23T01:46:33.6555085Z 
2022-11-23T01:46:33.6555167Z Generating XML reports...
2022-11-23T01:46:33.6555491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014611.xml
2022-11-23T01:46:33.6555707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6555978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6556101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6556379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6556514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6556551Z 
2022-11-23T01:46:33.6556623Z Running tests...
2022-11-23T01:46:33.6556814Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6557000Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6557005Z 
2022-11-23T01:46:33.6557193Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6557254Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6557259Z 
2022-11-23T01:46:33.6557328Z OK (skipped=1)
2022-11-23T01:46:33.6557359Z 
2022-11-23T01:46:33.6557441Z Generating XML reports...
2022-11-23T01:46:33.6557769Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014612.xml
2022-11-23T01:46:33.6557995Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6558272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6558396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6558673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6558807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6558812Z 
2022-11-23T01:46:33.6558871Z Running tests...
2022-11-23T01:46:33.6559065Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6559255Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6559262Z 
2022-11-23T01:46:33.6559449Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6559522Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6559526Z 
2022-11-23T01:46:33.6559595Z OK (skipped=1)
2022-11-23T01:46:33.6559599Z 
2022-11-23T01:46:33.6559684Z Generating XML reports...
2022-11-23T01:46:33.6560004Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014614.xml
2022-11-23T01:46:33.6560233Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6560493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6560612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6560887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6561018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6561023Z 
2022-11-23T01:46:33.6561089Z Running tests...
2022-11-23T01:46:33.6561276Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6561462Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6561467Z 
2022-11-23T01:46:33.6561653Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6561715Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6561731Z 
2022-11-23T01:46:33.6561787Z OK (skipped=1)
2022-11-23T01:46:33.6561791Z 
2022-11-23T01:46:33.6561874Z Generating XML reports...
2022-11-23T01:46:33.6562193Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014615.xml
2022-11-23T01:46:33.6562420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6562691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6562813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6563153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6563289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6563294Z 
2022-11-23T01:46:33.6563353Z Running tests...
2022-11-23T01:46:33.6563546Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6563734Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6563739Z 
2022-11-23T01:46:33.6563928Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6564042Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6564047Z 
2022-11-23T01:46:33.6564116Z OK (skipped=1)
2022-11-23T01:46:33.6564121Z 
2022-11-23T01:46:33.6564203Z Generating XML reports...
2022-11-23T01:46:33.6564529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014617.xml
2022-11-23T01:46:33.6564759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6565018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6565138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6565419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6565631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6565637Z 
2022-11-23T01:46:33.6565710Z Running tests...
2022-11-23T01:46:33.6565909Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6566105Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6566109Z 
2022-11-23T01:46:33.6566295Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6566358Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6566375Z 
2022-11-23T01:46:33.6566433Z OK (skipped=1)
2022-11-23T01:46:33.6566437Z 
2022-11-23T01:46:33.6566519Z Generating XML reports...
2022-11-23T01:46:33.6566840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014618.xml
2022-11-23T01:46:33.6567068Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6567339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6567462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6567741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6567876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6567881Z 
2022-11-23T01:46:33.6567940Z Running tests...
2022-11-23T01:46:33.6568132Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6568318Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6568323Z 
2022-11-23T01:46:33.6568514Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6568586Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6568591Z 
2022-11-23T01:46:33.6568659Z OK (skipped=1)
2022-11-23T01:46:33.6568663Z 
2022-11-23T01:46:33.6568746Z Generating XML reports...
2022-11-23T01:46:33.6569066Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014620.xml
2022-11-23T01:46:33.6569297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6569556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6569711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6569990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6570124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6570128Z 
2022-11-23T01:46:33.6570199Z Running tests...
2022-11-23T01:46:33.6570391Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6570590Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6570621Z 
2022-11-23T01:46:33.6570809Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6570880Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6570885Z 
2022-11-23T01:46:33.6570942Z OK (skipped=1)
2022-11-23T01:46:33.6570946Z 
2022-11-23T01:46:33.6571027Z Generating XML reports...
2022-11-23T01:46:33.6571354Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014621.xml
2022-11-23T01:46:33.6571581Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6571851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6571973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6572251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6572387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6572392Z 
2022-11-23T01:46:33.6572452Z Running tests...
2022-11-23T01:46:33.6572642Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6572844Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6572850Z 
2022-11-23T01:46:33.6573041Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6573113Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6573118Z 
2022-11-23T01:46:33.6573186Z OK (skipped=1)
2022-11-23T01:46:33.6573190Z 
2022-11-23T01:46:33.6573270Z Generating XML reports...
2022-11-23T01:46:33.6573587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014623.xml
2022-11-23T01:46:33.6573813Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6574074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6574195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6574473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6574609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6574614Z 
2022-11-23T01:46:33.6574686Z Running tests...
2022-11-23T01:46:33.6574967Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6575164Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.004s)
2022-11-23T01:46:33.6575169Z 
2022-11-23T01:46:33.6575360Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6575434Z Ran 1 test in 0.004s
2022-11-23T01:46:33.6575441Z 
2022-11-23T01:46:33.6575498Z OK (skipped=1)
2022-11-23T01:46:33.6575502Z 
2022-11-23T01:46:33.6575585Z Generating XML reports...
2022-11-23T01:46:33.6575903Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014624.xml
2022-11-23T01:46:33.6576133Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6576461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6576584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6576864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6576997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6577002Z 
2022-11-23T01:46:33.6577074Z Running tests...
2022-11-23T01:46:33.6577255Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6577478Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6577484Z 
2022-11-23T01:46:33.6577677Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6577751Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6577755Z 
2022-11-23T01:46:33.6577828Z OK (skipped=1)
2022-11-23T01:46:33.6577832Z 
2022-11-23T01:46:33.6577914Z Generating XML reports...
2022-11-23T01:46:33.6578236Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014625.xml
2022-11-23T01:46:33.6578466Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6578728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6578850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6579131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6579264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6579269Z 
2022-11-23T01:46:33.6579340Z Running tests...
2022-11-23T01:46:33.6579532Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6579717Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6579722Z 
2022-11-23T01:46:33.6579908Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6579980Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6579985Z 
2022-11-23T01:46:33.6580042Z OK (skipped=1)
2022-11-23T01:46:33.6580046Z 
2022-11-23T01:46:33.6580128Z Generating XML reports...
2022-11-23T01:46:33.6580446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014627.xml
2022-11-23T01:46:33.6580678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6580950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6581070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6581351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6581485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6581490Z 
2022-11-23T01:46:33.6581560Z Running tests...
2022-11-23T01:46:33.6581741Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6581928Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-11-23T01:46:33.6581934Z 
2022-11-23T01:46:33.6582124Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6582196Z Ran 1 test in 0.002s
2022-11-23T01:46:33.6582201Z 
2022-11-23T01:46:33.6582270Z OK (skipped=1)
2022-11-23T01:46:33.6582275Z 
2022-11-23T01:46:33.6582356Z Generating XML reports...
2022-11-23T01:46:33.6582705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014628.xml
2022-11-23T01:46:33.6582933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6583193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6583314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6583594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6583727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6583775Z 
2022-11-23T01:46:33.6583847Z Running tests...
2022-11-23T01:46:33.6584042Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6584222Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-11-23T01:46:33.6584228Z 
2022-11-23T01:46:33.6584415Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6584488Z Ran 1 test in 0.001s
2022-11-23T01:46:33.6584493Z 
2022-11-23T01:46:33.6584550Z OK (skipped=1)
2022-11-23T01:46:33.6584554Z 
2022-11-23T01:46:33.6584636Z Generating XML reports...
2022-11-23T01:46:33.6584959Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014630.xml
2022-11-23T01:46:33.6585188Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6585458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6585582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6585858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6585993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6586000Z 
2022-11-23T01:46:33.6586072Z Running tests...
2022-11-23T01:46:33.6586250Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6586427Z   test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-11-23T01:46:33.6586433Z 
2022-11-23T01:46:33.6586623Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6586700Z Ran 1 test in 0.004s
2022-11-23T01:46:33.6586704Z 
2022-11-23T01:46:33.6586775Z OK (skipped=1)
2022-11-23T01:46:33.6586780Z 
2022-11-23T01:46:33.6586865Z Generating XML reports...
2022-11-23T01:46:33.6587173Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123014631.xml
2022-11-23T01:46:33.6587400Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-11-23T01:46:33.6587659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:46:33.6587779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:46:33.6588055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:46:33.6588190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:46:33.6588195Z 
2022-11-23T01:46:33.6588267Z Running tests...
2022-11-23T01:46:33.6588458Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6588639Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-11-23T01:46:33.6588646Z 
2022-11-23T01:46:33.6588839Z ----------------------------------------------------------------------
2022-11-23T01:46:33.6588910Z Ran 1 test in 0.000s
2022-11-23T01:46:33.6588915Z 
2022-11-23T01:46:33.6588971Z OK (skipped=1)
2022-11-23T01:46:33.6588976Z 
2022-11-23T01:46:33.6589087Z Generating XML reports...
2022-11-23T01:46:33.6589378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123014633.xml
2022-11-23T01:46:33.6589383Z 
2022-11-23T01:46:33.6589769Z ##[endgroup]
2022-11-23T01:46:33.6590087Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_7dwezvtx)
2022-11-23T01:46:33.6590092Z 
2022-11-23T01:46:33.6590274Z Running distributed/test_c10d_pypg ... [2022-11-23 01:46:33.594936]
2022-11-23T01:46:33.6590624Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:46:33.595277]
2022-11-23T01:47:32.5770611Z 
2022-11-23T01:47:32.5771263Z Expand the folded group to see the log file of distributed/test_c10d_pypg
2022-11-23T01:47:32.5817325Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_7czp086p)
2022-11-23T01:47:32.5818147Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg
2022-11-23T01:47:32.5818530Z 
2022-11-23T01:47:32.5819140Z Running tests...
2022-11-23T01:47:32.5819654Z ----------------------------------------------------------------------
2022-11-23T01:47:32.5820260Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5821149Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5277
2022-11-23T01:47:32.5822097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5822493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5822927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5823276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5823584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5823871Z skip: Need at least 2 CUDA devices (1.767s)
2022-11-23T01:47:32.5824169Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5824577Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5288
2022-11-23T01:47:32.5825102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5825435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5825859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5826190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5826510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5826797Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5827077Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5827487Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5299
2022-11-23T01:47:32.5827992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5828322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5828732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5829074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5829597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5829886Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5830165Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5830570Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5310
2022-11-23T01:47:32.5831075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5831386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5831873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5832215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5832531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5832805Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5833114Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5833636Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5321
2022-11-23T01:47:32.5834149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5834460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5834889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5835231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5835535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5835824Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5836129Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5836653Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5332
2022-11-23T01:47:32.5837156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5837479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5837904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5838238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5838559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5838844Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5839138Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5839656Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5343
2022-11-23T01:47:32.5840171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5840494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5840917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5841248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5841566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5841851Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5842164Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5842696Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5354
2022-11-23T01:47:32.5843220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5843543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5843955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5844328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5844644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5844914Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5845203Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5845701Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5365
2022-11-23T01:47:32.5846222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5846540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5846963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5847309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5847629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5847903Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5848206Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5848634Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5376
2022-11-23T01:47:32.5849140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5849465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5849892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5850229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5850539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5850822Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5851128Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5851546Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5387
2022-11-23T01:47:32.5852066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5852388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5852810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5853141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5853463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5853751Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5854057Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5854490Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5398
2022-11-23T01:47:32.5855213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5855544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5855968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5856311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5856633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5856984Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5857275Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T01:47:32.5857679Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5409
2022-11-23T01:47:32.5858184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5858514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5858926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5859272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5859592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5859875Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5860233Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5420
2022-11-23T01:47:32.5860744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5861072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5861483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5861823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5862145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5862506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipylpyfh
2022-11-23T01:47:32.5862909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipylpyfh/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5863192Z ok (1.509s)
2022-11-23T01:47:32.5863510Z   test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5431
2022-11-23T01:47:32.5864010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5864333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5864757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5865090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5865514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5865872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7hzspns
2022-11-23T01:47:32.5866272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7hzspns/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5866650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:47:32.5866906Z ok (1.508s)
2022-11-23T01:47:32.5867277Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5442
2022-11-23T01:47:32.5867804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5868130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5868553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5868881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5869201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5869603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2qqoo6f
2022-11-23T01:47:32.5869987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2qqoo6f/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5870374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:47:32.5870634Z ok (1.508s)
2022-11-23T01:47:32.5870958Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5453
2022-11-23T01:47:32.5871469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5871795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5872222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5872555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5872880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5873479Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5874305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5875124Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5875935Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5876743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5877544Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5878033Z ok (1.509s)
2022-11-23T01:47:32.5878366Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5464
2022-11-23T01:47:32.5878887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5879213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5879625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5880001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5880322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5880599Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5880968Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5475
2022-11-23T01:47:32.5881490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5881817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5882273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5882614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5882935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5883226Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5883492Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5884008Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5486
2022-11-23T01:47:32.5884529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5884843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5885263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5885680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5886003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5886280Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5886570Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5886984Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5497
2022-11-23T01:47:32.5887491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5887816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5888239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5888580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5888882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5889167Z skip: Need at least 2 CUDA devices (1.408s)
2022-11-23T01:47:32.5889460Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5889865Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5508
2022-11-23T01:47:32.5890398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5890725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5891148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5891476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5891797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5892080Z skip: Need at least 2 CUDA devices (1.408s)
2022-11-23T01:47:32.5892368Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5892794Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5519
2022-11-23T01:47:32.5893298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5893621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5894033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5894369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5894691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5895137Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5895437Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5895964Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5530
2022-11-23T01:47:32.5896474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5896802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5897219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5897560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5897878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5898148Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5898459Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5898978Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5541
2022-11-23T01:47:32.5899494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5899807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5900233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5900570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5900875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5901156Z skip: Need at least 2 CUDA devices (1.508s)
2022-11-23T01:47:32.5901446Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5901977Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5552
2022-11-23T01:47:32.5902491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5902814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5903304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5903647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5903955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5904248Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5904537Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5905055Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5563
2022-11-23T01:47:32.5905634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5905961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5906386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5906716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5907033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5907318Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5907594Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5908007Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5574
2022-11-23T01:47:32.5908531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5908853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5909266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5909610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5909929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5910213Z skip: Need at least 2 CUDA devices (1.508s)
2022-11-23T01:47:32.5910503Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5910928Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5585
2022-11-23T01:47:32.5911445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5911761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5912186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5912532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5912853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5913127Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5913427Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5913851Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5596
2022-11-23T01:47:32.5914371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5914688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5915114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5915456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5915791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5916076Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5916376Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5916778Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5607
2022-11-23T01:47:32.5917265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5917619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5918048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5918375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5918696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5918987Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5919290Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T01:47:32.5919682Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5618
2022-11-23T01:47:32.5920188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5920511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5920934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5921264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5921581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5921869Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5922216Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5629
2022-11-23T01:47:32.5922730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5923054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5923481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5923813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5924131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5924500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoirj603s
2022-11-23T01:47:32.5924893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoirj603s/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5925171Z ok (1.509s)
2022-11-23T01:47:32.5925485Z   test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5640
2022-11-23T01:47:32.5926061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5926375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5926793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5927140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5927446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5927813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifuboj5i
2022-11-23T01:47:32.5928245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifuboj5i/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5928626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:47:32.5928867Z ok (1.508s)
2022-11-23T01:47:32.5929194Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5651
2022-11-23T01:47:32.5929713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5930067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5930482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5930822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5931145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5931500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqejpfdc
2022-11-23T01:47:32.5931901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqejpfdc/_remote_module_non_scriptable.py
2022-11-23T01:47:32.5932281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:47:32.5932536Z ok (1.508s)
2022-11-23T01:47:32.5932853Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5662
2022-11-23T01:47:32.5933372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5933694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5934104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5934446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5934765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5935499Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5936330Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5937133Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5937940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5938748Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5939609Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:47:32.5940081Z ok (1.508s)
2022-11-23T01:47:32.5940412Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5673
2022-11-23T01:47:32.5940964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5941295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5941723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5942070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5942380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5942666Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5943033Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5684
2022-11-23T01:47:32.5943553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:47:32.5943870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:47:32.5944294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:47:32.5944634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:47:32.5944939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:47:32.5945226Z skip: Need at least 2 CUDA devices (1.509s)
2022-11-23T01:47:32.5945360Z 
2022-11-23T01:47:32.5945557Z ----------------------------------------------------------------------
2022-11-23T01:47:32.5945797Z Ran 38 tests in 57.393s
2022-11-23T01:47:32.5945897Z 
2022-11-23T01:47:32.5945968Z OK (skipped=30)
2022-11-23T01:47:32.5946074Z 
2022-11-23T01:47:32.5946157Z Generating XML reports...
2022-11-23T01:47:32.5946608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123014634.xml
2022-11-23T01:47:32.5947190Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123014634.xml
2022-11-23T01:47:32.5947447Z 
2022-11-23T01:47:32.5947735Z ##[endgroup]
2022-11-23T01:47:32.5948155Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_7czp086p)
2022-11-23T01:47:32.5948390Z 
2022-11-23T01:47:32.5948595Z Running distributed/rpc/test_tensorpipe_agent ... [2022-11-23 01:47:32.577914]
2022-11-23T01:47:32.5949136Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:47:32.578378]
2022-11-23T02:20:41.9093444Z 
2022-11-23T02:20:41.9093885Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2022-11-23T02:20:41.9096158Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_x2yvtp4u)
2022-11-23T02:20:41.9126974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1wunsza
2022-11-23T02:20:41.9127730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1wunsza/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9200132Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]>
2022-11-23T02:20:41.9201189Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest)
2022-11-23T02:20:41.9201717Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest)
2022-11-23T02:20:41.9202478Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest)
2022-11-23T02:20:41.9203382Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest)
2022-11-23T02:20:41.9205256Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_inside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]>
2022-11-23T02:20:41.9256117Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-11-23T02:20:41.9256768Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-11-23T02:20:41.9257474Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-11-23T02:20:41.9258019Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-11-23T02:20:41.9266932Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistAutogradTest testMethod=test_async_dist_autograd>, <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]>
2022-11-23T02:20:41.9275711Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9276184Z test_autograd_context (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9276705Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9296085Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9296679Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9297194Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9297788Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9298336Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9298845Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9299449Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9299999Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9300538Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9301136Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9301727Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9302311Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9302808Z test_backward_rref (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9303500Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9304062Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9304592Z test_backward_simple (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9305138Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9305728Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9306259Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9306843Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9307455Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9307975Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9308558Z test_backward_without_context (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9309087Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9309663Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9310243Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9310829Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9311359Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9311856Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9312170Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9312475Z test_debug_info (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9312853Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9313237Z test_error_in_context (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9313630Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9314103Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9314537Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9314986Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9315439Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9315886Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9316314Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9316641Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9316974Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9317279Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9317591Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9317901Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9318198Z test_multiple_backward (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9318492Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9318821Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9319131Z test_nested_context (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9319399Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9319688Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9320010Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9320337Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9320651Z test_post_hooks (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9320942Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9321305Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9321592Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9321885Z test_trainer_ps (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9322194Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9322498Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:41.9323138Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]>
2022-11-23T02:20:41.9323822Z test_dist_optim (__main__.TensorPipeDistOptimizerTest)
2022-11-23T02:20:41.9324119Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest)
2022-11-23T02:20:41.9327682Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest)
2022-11-23T02:20:41.9328309Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest)
2022-11-23T02:20:41.9329498Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitDistAutogradTest testMethod=test_dist_backward>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]>
2022-11-23T02:20:41.9330193Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest)
2022-11-23T02:20:41.9330498Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest)
2022-11-23T02:20:41.9330808Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest)
2022-11-23T02:20:41.9331154Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest)
2022-11-23T02:20:41.9339563Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]>
2022-11-23T02:20:41.9352623Z test_add_done_callback (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9353839Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9354995Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9357286Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9358272Z test_async_function_remote (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9359223Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9360352Z test_async_function_simple (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9362192Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9362767Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9364170Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9365880Z test_async_script_throw (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9366439Z test_async_script_udf (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9369069Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9370640Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9371091Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9371561Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9372069Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9372464Z test_callback_chain (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9372849Z test_callback_simple (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9373219Z test_callback_with_exception (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9373619Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9373955Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9374398Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9377023Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9377442Z test_future_python_annotation (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9377955Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9378376Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9378860Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9379321Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9379760Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9380166Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9380614Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9381110Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9381601Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9382042Z test_remote_script_module (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9382464Z test_remote_script_throw (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9382872Z test_remote_script_udf (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9383321Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9383866Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9384360Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9384817Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9385293Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9385699Z test_rref_is_owner (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9386135Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9386543Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9386972Z test_rref_local_value (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9387454Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9387965Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9388543Z test_torchscript_function (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9389063Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9389610Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9390099Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9390594Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9391068Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest)
2022-11-23T02:20:41.9391692Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeParameterServerTest testMethod=test_batch_updating_parameter_server>]>
2022-11-23T02:20:41.9392357Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest)
2022-11-23T02:20:41.9393021Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeReinforcementLearningRpcTest testMethod=test_rl_rpc>]>
2022-11-23T02:20:41.9393671Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest)
2022-11-23T02:20:41.9396151Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRemoteModuleTest testMethod=test_bad_module>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]>
2022-11-23T02:20:41.9398638Z test_bad_module (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9399113Z test_forward_async (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9399582Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9400081Z test_forward_sync (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9400560Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9401031Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9401529Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9402052Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9402640Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9403170Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9403745Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9404300Z test_train_eval (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9404770Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest)
2022-11-23T02:20:41.9429326Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRpcTest testMethod=test_add>, <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_legacy>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_custom_exception_throw_during_reconstruction>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]>
2022-11-23T02:20:41.9450134Z test_add (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9450595Z test_add_done_callback (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9451043Z test_add_with_id (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9451482Z test_all_gather (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9451910Z test_all_gather_timeout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9452361Z test_async_add (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9452801Z test_async_class_method (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9453270Z test_async_class_method_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9453765Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9454199Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9454610Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9455038Z test_async_function_chained (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9455404Z test_async_function_chained_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9455818Z test_async_function_multi_chained (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9456248Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9456704Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9457148Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9457559Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9458048Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9458521Z test_async_function_nested (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9458975Z test_async_function_nested_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9459411Z test_async_function_raise (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9459846Z test_async_function_raise_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9460267Z test_async_function_raise_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9460668Z test_async_function_simple (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9461078Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9461517Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9461913Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9462301Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9462737Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9463167Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9463608Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9464079Z test_async_record_function_legacy (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9464539Z test_async_static_method (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9464994Z test_async_static_method_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9465420Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9465890Z test_builtin_remote_ret (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9466460Z test_builtin_remote_self (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9466898Z test_call_method_on_rref (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9467341Z test_callback_chain (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9467762Z test_callback_in_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9468169Z test_callback_multi (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9468514Z test_callback_none (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9468940Z test_callback_simple (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9469363Z test_callback_with_error (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9469760Z test_callback_with_ret (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9470198Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9470611Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9471015Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9471431Z test_custom_exception_throw_during_reconstruction (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9471760Z test_deadlock (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9472057Z test_debug_info (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9472382Z test_default_timeout_used (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9472702Z test_disable_gil_profiling (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9473054Z test_dist_init_decorator (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9473432Z test_duplicate_name (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9473814Z test_duplicate_name_2 (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9474235Z test_expected_src (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9474639Z test_function_not_on_callee (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9474994Z test_future_done (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9475320Z test_future_done_exception (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9475732Z test_future_in_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9475999Z test_future_nested_callback (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9476319Z test_future_wait_twice (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9476639Z test_get_worker_infos (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9476914Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9477211Z test_handle_send_exceptions (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9477476Z test_ignore_rref_leak (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9477754Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9478042Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9478297Z test_init_rpc_twice (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9478529Z test_int_callee (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9478774Z test_invalid_names (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9479029Z test_local_rref_no_fork (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9479284Z test_local_shutdown (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9479537Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9479808Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9480071Z test_mark_future_twice (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9480323Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9480599Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9480871Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9481108Z test_multi_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9481361Z test_my_parameter_server (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9481616Z test_nested_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9481861Z test_nested_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9482092Z test_nested_rref (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9482343Z test_nested_rref_stress (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9482659Z test_non_cont_tensors (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9482964Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9483267Z test_nonzero (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9483514Z test_owner_equality (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9483758Z test_owner_rref_backward (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9484018Z test_pass_local_rrefs (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9484278Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9484530Z test_pickle_future (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9484819Z test_profiler_export_trace (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9485177Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9485496Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9485785Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9486057Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9486331Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9486602Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9486912Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9487217Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9487519Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9487809Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9488122Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9488430Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9488733Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9489016Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9489311Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9489610Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9489907Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9490217Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9490526Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9490832Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9491127Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9491432Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9491736Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9492023Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9492323Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9492603Z test_py_built_in (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9492860Z test_py_class_constructor (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9493120Z test_py_class_instance_method (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9493385Z test_py_class_method (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9493649Z test_py_class_static_method (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9493905Z test_py_function_exception (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9494177Z test_py_multi_async_call (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9494437Z test_py_nested_pickle (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9494684Z test_py_no_return_result (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9495081Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9495436Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9495712Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9495948Z test_py_rref_args (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9496210Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9496467Z test_py_tensors (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9496716Z test_py_tensors_in_container (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9496993Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9497260Z test_py_udf_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9497544Z test_py_user_defined (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9497851Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9498138Z test_reinit (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9498388Z test_remote_same_worker (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9498628Z test_remote_throw (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9498894Z test_remote_with_exception (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9499154Z test_return_future (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9499397Z test_return_future_async (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9499665Z test_return_future_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9499927Z test_return_local_rrefs (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9500172Z test_rpc_barrier_all (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9500441Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9500724Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9500990Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9501255Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9501565Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9501885Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9502146Z test_rpc_return_rref (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9502393Z test_rpc_timeouts (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9502652Z test_rref_context_debug_info (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9502900Z test_rref_forward_chain (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9503153Z test_rref_get_future (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9503394Z test_rref_leak (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9503636Z test_rref_proxy_class (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9503884Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9504142Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9504397Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9504636Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9504894Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9505171Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9505412Z test_rref_str (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9505650Z test_rref_timeout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9505897Z test_rref_type_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9506160Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9506414Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9506682Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9506949Z test_rref_type_slow_init (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9507207Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9507490Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9507751Z test_scalar_add (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9507975Z test_self_add (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9508264Z test_self_py_udf_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9508537Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9508802Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9509084Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9509372Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9509631Z test_send_to_rank (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9509883Z test_server_process_global_profiler (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9510197Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9510463Z test_shutdown_errors (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9510713Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9510974Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9511240Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9511495Z test_stress_light_rpc (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9511743Z test_use_rpc_pickler (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9512004Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9512266Z test_user_rref_backward (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9512515Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9512786Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9513037Z test_wait_all (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9513281Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9513555Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9513836Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9514104Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9514363Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9514632Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9514891Z test_wait_all_timeout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9515144Z test_wait_all_with_exception (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9515420Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9515697Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9515954Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9516227Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9516482Z test_worker_id (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9516741Z test_worker_info_pickle (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9516991Z test_world_size_one (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9517233Z test_wrong_types (__main__.TensorPipeRpcTest)
2022-11-23T02:20:41.9521814Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_different_dtypes_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]>
2022-11-23T02:20:41.9526526Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9526946Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9527353Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9527738Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9528132Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9528513Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9528901Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9529296Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9529689Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9530071Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9530451Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9530856Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9531263Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9531679Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9532077Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9532476Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9532912Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9533307Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9533706Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9534117Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9534531Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9535075Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9593507Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9593923Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9594334Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9594748Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9595176Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9595611Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9596019Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9596409Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9596778Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-11-23T02:20:41.9601632Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_and_static_init_rpc_together>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_init_rpc_without_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_dynamic_rpc_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]>
2022-11-23T02:20:41.9606347Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9606687Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9607047Z test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9607443Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9607861Z test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9608224Z test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9608578Z test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9608969Z test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9609333Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9609682Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9610035Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9610386Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9610708Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9611043Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9611381Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9611697Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9612025Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9612359Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9612683Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9613020Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9613354Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9613696Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9614041Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9614394Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9614752Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9615259Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9615598Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9615950Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9616313Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9616670Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9617042Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9617434Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9617762Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9618106Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9618472Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9618848Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9619177Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9619521Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9619865Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-11-23T02:20:41.9620567Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_create_remote_module_from_module_rref>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]>
2022-11-23T02:20:41.9621263Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-11-23T02:20:41.9621654Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-11-23T02:20:41.9622064Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-11-23T02:20:41.9622671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9622995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9623432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9623780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9624117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmks14u8z
2022-11-23T02:20:41.9624518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmks14u8z/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9624989Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9625199Z 
2022-11-23T02:20:41.9625273Z Running tests...
2022-11-23T02:20:41.9625566Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9626433Z   test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.251s)
2022-11-23T02:20:41.9626883Z 
2022-11-23T02:20:41.9627079Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9627315Z Ran 1 test in 0.252s
2022-11-23T02:20:41.9627427Z 
2022-11-23T02:20:41.9627484Z OK (skipped=1)
2022-11-23T02:20:41.9627590Z 
2022-11-23T02:20:41.9627673Z Generating XML reports...
2022-11-23T02:20:41.9628191Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014735.xml
2022-11-23T02:20:41.9628756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9629076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9629501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9629846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9630172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5zswrpv
2022-11-23T02:20:41.9630597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5zswrpv/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9631059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9631258Z 
2022-11-23T02:20:41.9631327Z Running tests...
2022-11-23T02:20:41.9631614Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9632485Z   test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T02:20:41.9632944Z 
2022-11-23T02:20:41.9633133Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9633366Z Ran 1 test in 0.252s
2022-11-23T02:20:41.9633474Z 
2022-11-23T02:20:41.9633532Z OK (skipped=1)
2022-11-23T02:20:41.9633633Z 
2022-11-23T02:20:41.9633717Z Generating XML reports...
2022-11-23T02:20:41.9634197Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014737.xml
2022-11-23T02:20:41.9634755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9635066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9635491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9635836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9636164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vgjtwkk
2022-11-23T02:20:41.9636562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vgjtwkk/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9637029Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9637236Z 
2022-11-23T02:20:41.9637307Z Running tests...
2022-11-23T02:20:41.9637592Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9638008Z   test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5736
2022-11-23T02:20:41.9638421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5737
2022-11-23T02:20:41.9638745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5738
2022-11-23T02:20:41.9639053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5739
2022-11-23T02:20:41.9639584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9639914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9640384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9640726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9641115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4me76_y
2022-11-23T02:20:41.9641516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4me76_y/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9641990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9642309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9642728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9643100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9643440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3crigoc
2022-11-23T02:20:41.9643829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3crigoc/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9644310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9644619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9645115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9645453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9645789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2mmex0m
2022-11-23T02:20:41.9646174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2mmex0m/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9646662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9646983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9647391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9647726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9648062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1gn8elx
2022-11-23T02:20:41.9648461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1gn8elx/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9648823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:41.9649166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:41.9649509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:41.9649841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:41.9650197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:41.9650561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:41.9650926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:41.9651275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:41.9651762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9652278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9652799Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9653297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9653717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9654070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9654489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9654826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9655277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9655627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9656018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9656360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9656612Z ok (2.679s)
2022-11-23T02:20:41.9656713Z 
2022-11-23T02:20:41.9656920Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9657146Z Ran 1 test in 2.679s
2022-11-23T02:20:41.9657256Z 
2022-11-23T02:20:41.9657315Z OK
2022-11-23T02:20:41.9657403Z 
2022-11-23T02:20:41.9657484Z Generating XML reports...
2022-11-23T02:20:41.9657959Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014739.xml
2022-11-23T02:20:41.9658525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9658853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9659279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9659610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9659952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93g16yrm
2022-11-23T02:20:41.9660349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93g16yrm/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9660802Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9661009Z 
2022-11-23T02:20:41.9661076Z Running tests...
2022-11-23T02:20:41.9661373Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9661789Z   test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5991
2022-11-23T02:20:41.9662194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5992
2022-11-23T02:20:41.9662514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5993
2022-11-23T02:20:41.9662834Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5994
2022-11-23T02:20:41.9663280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9663595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9664022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9664365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9664694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7gd_r8h
2022-11-23T02:20:41.9665207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7gd_r8h/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9665716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9666036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9666506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9666850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9667184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvoo_7si
2022-11-23T02:20:41.9667568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvoo_7si/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9668037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9668343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9668784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9669109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9669444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyio_sh4s
2022-11-23T02:20:41.9669837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyio_sh4s/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9670308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9670614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9671022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9671347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9671666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv49jhggc
2022-11-23T02:20:41.9672051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv49jhggc/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9672412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:41.9672748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:41.9673072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:41.9673401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:41.9673762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:41.9674114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:41.9674463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:41.9674815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:41.9675290Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9675794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9676330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9676837Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:41.9677213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9677548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9677888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9678222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9678456Z ok (2.675s)
2022-11-23T02:20:41.9678550Z 
2022-11-23T02:20:41.9678742Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9678999Z Ran 1 test in 2.676s
2022-11-23T02:20:41.9679105Z 
2022-11-23T02:20:41.9679159Z OK
2022-11-23T02:20:41.9679236Z 
2022-11-23T02:20:41.9679311Z Generating XML reports...
2022-11-23T02:20:41.9679789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014744.xml
2022-11-23T02:20:41.9680337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9680652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9681091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9681424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9681759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyr2iku92
2022-11-23T02:20:41.9682141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyr2iku92/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9682600Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9682799Z 
2022-11-23T02:20:41.9682863Z Running tests...
2022-11-23T02:20:41.9683157Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9683560Z   test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6258
2022-11-23T02:20:41.9683976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6259
2022-11-23T02:20:41.9684297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6260
2022-11-23T02:20:41.9684606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6261
2022-11-23T02:20:41.9684983Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 6262
2022-11-23T02:20:41.9685291Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 6263
2022-11-23T02:20:41.9685729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9686046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9686461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9686791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9687125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkcz107e
2022-11-23T02:20:41.9687508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkcz107e/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9687981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9688298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9688707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9689038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9689403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpwqqm_k
2022-11-23T02:20:41.9689800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpwqqm_k/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9690160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:20:41.9690613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9690924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9691370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9691696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9692042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9gb794yd
2022-11-23T02:20:41.9692444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9gb794yd/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9692816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:41.9693259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9693612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9694032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9694367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9694696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uyp26te
2022-11-23T02:20:41.9695345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9695664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9696078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9696403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9696729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ahe46e1
2022-11-23T02:20:41.9697113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ahe46e1/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9697492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uyp26te/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9697963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9698270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9698676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9698997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9699326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu33m55ng
2022-11-23T02:20:41.9699707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu33m55ng/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9700088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:20:41.9700421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:41.9700751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:41.9701088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:41.9701527Z 2022-11-23 01:47:51,581 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-11-23T02:20:41.9702037Z 2022-11-23 01:47:51,582 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9702510Z 2022-11-23 01:47:51,601 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-11-23T02:20:41.9703009Z 2022-11-23 01:47:51,601 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9703473Z 2022-11-23 01:47:51,611 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-11-23T02:20:41.9704025Z 2022-11-23 01:47:51,611 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9704498Z 2022-11-23 01:47:51,621 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-11-23T02:20:41.9704990Z 2022-11-23 01:47:51,621 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9705467Z 2022-11-23 01:47:51,622 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-11-23T02:20:41.9705899Z 2022-11-23 01:47:51,621 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-11-23T02:20:41.9706291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:41.9706649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:41.9707011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:41.9707366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T02:20:41.9707720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T02:20:41.9708072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:41.9708588Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9709098Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9709605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9710107Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9710604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9711090Z 2022-11-23 01:47:51,674 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-11-23T02:20:41.9711576Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9712063Z 2022-11-23 01:47:51,675 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-11-23T02:20:41.9712430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T02:20:41.9712867Z 2022-11-23 01:47:51,676 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-11-23T02:20:41.9713223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T02:20:41.9713700Z 2022-11-23 01:47:51,684 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-11-23T02:20:41.9714181Z 2022-11-23 01:47:51,684 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-11-23T02:20:41.9714615Z 2022-11-23 01:47:51,707 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-11-23T02:20:41.9715196Z 2022-11-23 01:47:51,708 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-11-23T02:20:41.9715607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:20:41.9716155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:20:41.9716756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:20:41.9717313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:20:41.9718083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9718946Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9719707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9720651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9721433Z 2022-11-23 01:47:51,860 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net.
2022-11-23T02:20:41.9722554Z 2022-11-23 01:47:51,860 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net.
2022-11-23T02:20:41.9723391Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9724288Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9725141Z 2022-11-23 01:47:51,870 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net.
2022-11-23T02:20:41.9725859Z 2022-11-23 01:47:51,870 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net.
2022-11-23T02:20:41.9726853Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9727549Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9728185Z 2022-11-23 01:47:51,903 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9728745Z 2022-11-23 01:47:51,903 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-11-23T02:20:41.9729393Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9729807Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9730241Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9730775Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-11-23T02:20:41.9731409Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9731814Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9732242Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9732778Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-11-23T02:20:41.9733414Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9733820Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9734355Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9735016Z 2022-11-23 01:47:51,904 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-11-23T02:20:41.9735725Z 2022-11-23 01:47:51,949 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9736025Z         [-1.,  1.],
2022-11-23T02:20:41.9736283Z         [-1., -1.],
2022-11-23T02:20:41.9736657Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9737057Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9737304Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9737561Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:41.9737777Z         [-16.,  16.]])}
2022-11-23T02:20:41.9738194Z 2022-11-23 01:47:51,949 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9738503Z         [-1., -1.],
2022-11-23T02:20:41.9738680Z         [-1.,  1.],
2022-11-23T02:20:41.9739045Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9739447Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9739697Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9739947Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:41.9740157Z         [ 16., -16.]])}
2022-11-23T02:20:41.9740572Z 2022-11-23 01:47:51,949 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9740878Z         [ 1., -1.],
2022-11-23T02:20:41.9741024Z         [ 1.,  1.],
2022-11-23T02:20:41.9741397Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9741792Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9742037Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9742354Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:41.9742578Z         [  0., -16.]])}
2022-11-23T02:20:41.9742993Z 2022-11-23 01:47:51,952 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9743263Z         [ 1.,  1.],
2022-11-23T02:20:41.9743442Z         [ 1., -1.],
2022-11-23T02:20:41.9743814Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9744202Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9744450Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9744661Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:41.9744854Z         [ 0., 16.]])}
2022-11-23T02:20:41.9745103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9745453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9745804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9746142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9746694Z 2022-11-23 01:47:51,978 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9747005Z         [-1.,  1.],
2022-11-23T02:20:41.9747179Z         [-1., -1.],
2022-11-23T02:20:41.9747550Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9747946Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9748196Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9748443Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:41.9748667Z         [-16.,  16.]])}
2022-11-23T02:20:41.9749106Z 2022-11-23 01:47:51,978 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9749373Z         [ 1.,  1.],
2022-11-23T02:20:41.9749584Z         [ 1., -1.],
2022-11-23T02:20:41.9749949Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9750338Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9750580Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9750774Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:41.9750956Z         [ 0., 16.]])}
2022-11-23T02:20:41.9751363Z 2022-11-23 01:47:51,978 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9751669Z         [-1., -1.],
2022-11-23T02:20:41.9751844Z         [-1.,  1.],
2022-11-23T02:20:41.9752307Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9752946Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9753327Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9753716Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:41.9754076Z         [ 16., -16.]])}
2022-11-23T02:20:41.9754828Z 2022-11-23 01:47:51,981 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9755137Z         [ 1., -1.],
2022-11-23T02:20:41.9755280Z         [ 1.,  1.],
2022-11-23T02:20:41.9755654Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9756055Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9756302Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9756546Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:41.9756764Z         [  0., -16.]])}
2022-11-23T02:20:41.9757175Z 2022-11-23 01:47:52,012 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9757472Z         [ 1., -1.],
2022-11-23T02:20:41.9757621Z         [ 1.,  1.],
2022-11-23T02:20:41.9757989Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9758372Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9758620Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9758872Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:41.9759094Z         [  0., -16.]])}
2022-11-23T02:20:41.9759494Z 2022-11-23 01:47:52,012 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9759763Z         [ 1.,  1.],
2022-11-23T02:20:41.9759938Z         [ 1., -1.],
2022-11-23T02:20:41.9760364Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9760763Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9761083Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9761277Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:41.9761455Z         [ 0., 16.]])}
2022-11-23T02:20:41.9761871Z 2022-11-23 01:47:52,012 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9762220Z         [-1., -1.],
2022-11-23T02:20:41.9762394Z         [-1.,  1.],
2022-11-23T02:20:41.9762765Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9763165Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9763411Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9763665Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:41.9763885Z         [ 16., -16.]])}
2022-11-23T02:20:41.9764302Z 2022-11-23 01:47:52,015 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9764596Z         [-1.,  1.],
2022-11-23T02:20:41.9764775Z         [-1., -1.],
2022-11-23T02:20:41.9765209Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9765602Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9765853Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9766102Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:41.9766313Z         [-16.,  16.]])}
2022-11-23T02:20:41.9766672Z 2022-11-23 01:47:52,020 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-11-23T02:20:41.9767111Z 2022-11-23 01:47:52,021 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-11-23T02:20:41.9767579Z 2022-11-23 01:47:52,021 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-11-23T02:20:41.9767997Z 2022-11-23 01:47:52,022 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-11-23T02:20:41.9768426Z 2022-11-23 01:47:52,023 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-11-23T02:20:41.9768673Z ok (3.886s)
2022-11-23T02:20:41.9768771Z 
2022-11-23T02:20:41.9768964Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9769187Z Ran 1 test in 3.886s
2022-11-23T02:20:41.9769295Z 
2022-11-23T02:20:41.9769353Z OK
2022-11-23T02:20:41.9769441Z 
2022-11-23T02:20:41.9769521Z Generating XML reports...
2022-11-23T02:20:41.9770026Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014748.xml
2022-11-23T02:20:41.9770608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9770931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9771349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9771693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9772030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktqwxv12
2022-11-23T02:20:41.9772424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktqwxv12/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9772907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9773114Z 
2022-11-23T02:20:41.9773186Z Running tests...
2022-11-23T02:20:41.9773481Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9774027Z   test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6672
2022-11-23T02:20:41.9774656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6673
2022-11-23T02:20:41.9775267Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6674
2022-11-23T02:20:41.9775841Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6675
2022-11-23T02:20:41.9776412Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 6676
2022-11-23T02:20:41.9776845Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 6677
2022-11-23T02:20:41.9777312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9777638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9778054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9778393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9778728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_ieg1yk
2022-11-23T02:20:41.9779115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_ieg1yk/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9779588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9779907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9780324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9780669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9781003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac4gb608
2022-11-23T02:20:41.9781413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac4gb608/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9781910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9782221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9782641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9782972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9783304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpft4q6io5
2022-11-23T02:20:41.9783687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpft4q6io5/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9784051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:41.9784390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:41.9784829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9785145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9785558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9785894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9786219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjymwets2
2022-11-23T02:20:41.9786672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjymwets2/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9787150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9787457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9787872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9788205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9788535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vrrujeh
2022-11-23T02:20:41.9788981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vrrujeh/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9789460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9789779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9790192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9790517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9790848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtm0xwzp
2022-11-23T02:20:41.9791238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtm0xwzp/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9791601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:20:41.9791941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:20:41.9792275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:41.9792607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:41.9793048Z 2022-11-23 01:47:57,297 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-11-23T02:20:41.9793558Z 2022-11-23 01:47:57,297 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9794041Z 2022-11-23 01:47:57,307 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-11-23T02:20:41.9794545Z 2022-11-23 01:47:57,307 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9795005Z 2022-11-23 01:47:57,317 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-11-23T02:20:41.9795443Z 2022-11-23 01:47:57,317 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-11-23T02:20:41.9795938Z 2022-11-23 01:47:57,317 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9796464Z 2022-11-23 01:47:57,317 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9796932Z 2022-11-23 01:47:57,327 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-11-23T02:20:41.9797380Z 2022-11-23 01:47:57,327 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-11-23T02:20:41.9797737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:41.9798102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:41.9798448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:41.9798837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:41.9799192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T02:20:41.9799531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T02:20:41.9800023Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9800534Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9801071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9801576Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9802084Z 2022-11-23 01:47:57,390 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-11-23T02:20:41.9802569Z 2022-11-23 01:47:57,390 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-11-23T02:20:41.9803053Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9803558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9804057Z 2022-11-23 01:47:57,390 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-11-23T02:20:41.9804544Z 2022-11-23 01:47:57,390 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-11-23T02:20:41.9804991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T02:20:41.9805453Z 2022-11-23 01:47:57,400 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-11-23T02:20:41.9805817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T02:20:41.9806255Z 2022-11-23 01:47:57,431 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-11-23T02:20:41.9806685Z 2022-11-23 01:47:57,432 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-11-23T02:20:41.9807047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:20:41.9807409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:20:41.9807770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:20:41.9808120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:20:41.9808604Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9809188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9809988Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9810878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9811762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9812650Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9813642Z 2022-11-23 01:47:57,582 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9814534Z 2022-11-23 01:47:57,582 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9815511Z 2022-11-23 01:47:57,584 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9816396Z 2022-11-23 01:47:57,584 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9817264Z 2022-11-23 01:47:57,584 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9818220Z 2022-11-23 01:47:57,584 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9819093Z 2022-11-23 01:47:57,591 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9819975Z 2022-11-23 01:47:57,591 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9821118Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9821903Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9822847Z 2022-11-23 01:47:57,615 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9824071Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9824860Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9825807Z 2022-11-23 01:47:57,615 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9827015Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9827810Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9828757Z 2022-11-23 01:47:57,616 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9829971Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9830763Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9831722Z 2022-11-23 01:47:57,616 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9832743Z 2022-11-23 01:47:57,665 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9833299Z         [-1., -1.],
2022-11-23T02:20:41.9833636Z         [-1.,  1.],
2022-11-23T02:20:41.9834344Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9835093Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9835558Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9835928Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9836267Z         [0., 0.]])}
2022-11-23T02:20:41.9837111Z 2022-11-23 01:47:57,665 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9837681Z         [ 1., -1.],
2022-11-23T02:20:41.9837961Z         [ 1.,  1.],
2022-11-23T02:20:41.9838649Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9839393Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9839857Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9840260Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9840593Z         [0., 0.]])}
2022-11-23T02:20:41.9841371Z 2022-11-23 01:47:57,665 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9841929Z         [-1.,  1.],
2022-11-23T02:20:41.9842240Z         [-1., -1.],
2022-11-23T02:20:41.9842930Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9843661Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9844114Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9844483Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9844825Z         [0., 0.]])}
2022-11-23T02:20:41.9845662Z 2022-11-23 01:47:57,665 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9846175Z         [ 1.,  1.],
2022-11-23T02:20:41.9846506Z         [ 1., -1.],
2022-11-23T02:20:41.9847200Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9847925Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9848389Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9848751Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9849083Z         [0., 0.]])}
2022-11-23T02:20:41.9849550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9850208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9850859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9851488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9852445Z 2022-11-23 01:47:57,698 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9852958Z         [ 1.,  1.],
2022-11-23T02:20:41.9853260Z         [ 1., -1.],
2022-11-23T02:20:41.9853915Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9854652Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9855223Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9855579Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9855925Z         [0., 0.]])}
2022-11-23T02:20:41.9856707Z 2022-11-23 01:47:57,698 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9857274Z         [ 1., -1.],
2022-11-23T02:20:41.9857561Z         [ 1.,  1.],
2022-11-23T02:20:41.9858272Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9859020Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9859556Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9859927Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9860268Z         [0., 0.]])}
2022-11-23T02:20:41.9861022Z 2022-11-23 01:47:57,698 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9861579Z         [-1.,  1.],
2022-11-23T02:20:41.9861919Z         [-1., -1.],
2022-11-23T02:20:41.9862597Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9863394Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9863859Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9864227Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9864560Z         [0., 0.]])}
2022-11-23T02:20:41.9865352Z 2022-11-23 01:47:57,701 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9865921Z         [-1., -1.],
2022-11-23T02:20:41.9866245Z         [-1.,  1.],
2022-11-23T02:20:41.9866948Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9867697Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9868135Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9868502Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9868847Z         [0., 0.]])}
2022-11-23T02:20:41.9869625Z 2022-11-23 01:47:57,722 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9870194Z         [-1., -1.],
2022-11-23T02:20:41.9870529Z         [-1.,  1.],
2022-11-23T02:20:41.9871235Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9871968Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9872431Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9872797Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9873128Z         [0., 0.]])}
2022-11-23T02:20:41.9873900Z 2022-11-23 01:47:57,722 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9874480Z         [ 1., -1.],
2022-11-23T02:20:41.9874760Z         [ 1.,  1.],
2022-11-23T02:20:41.9875447Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9876192Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9876655Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9877012Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9877349Z         [0., 0.]])}
2022-11-23T02:20:41.9878112Z 2022-11-23 01:47:57,722 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9878624Z         [ 1.,  1.],
2022-11-23T02:20:41.9878946Z         [ 1., -1.],
2022-11-23T02:20:41.9879630Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9880375Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9880826Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9881193Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9881531Z         [0., 0.]])}
2022-11-23T02:20:41.9882342Z 2022-11-23 01:47:57,722 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9882913Z         [-1.,  1.],
2022-11-23T02:20:41.9883240Z         [-1., -1.],
2022-11-23T02:20:41.9883928Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9884658Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9885192Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9885607Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-11-23T02:20:41.9885939Z         [0., 0.]])}
2022-11-23T02:20:41.9886594Z 2022-11-23 01:47:57,730 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-11-23T02:20:41.9887424Z 2022-11-23 01:47:57,731 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-11-23T02:20:41.9888233Z 2022-11-23 01:47:57,731 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-11-23T02:20:41.9889004Z 2022-11-23 01:47:57,732 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-11-23T02:20:41.9889696Z 2022-11-23 01:47:57,733 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-11-23T02:20:41.9890149Z ok (3.781s)
2022-11-23T02:20:41.9890343Z 
2022-11-23T02:20:41.9890625Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9890867Z Ran 1 test in 3.782s
2022-11-23T02:20:41.9890976Z 
2022-11-23T02:20:41.9891037Z OK
2022-11-23T02:20:41.9891125Z 
2022-11-23T02:20:41.9891210Z Generating XML reports...
2022-11-23T02:20:41.9891717Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014754.xml
2022-11-23T02:20:41.9892303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9892631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9893045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9893391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9893738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpau6y9wgh
2022-11-23T02:20:41.9894137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpau6y9wgh/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9894590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:41.9894798Z 
2022-11-23T02:20:41.9894979Z Running tests...
2022-11-23T02:20:41.9895304Z ----------------------------------------------------------------------
2022-11-23T02:20:41.9895749Z   test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7083
2022-11-23T02:20:41.9896171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7084
2022-11-23T02:20:41.9896495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7085
2022-11-23T02:20:41.9896818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7086
2022-11-23T02:20:41.9897125Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 7087
2022-11-23T02:20:41.9897440Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 7088
2022-11-23T02:20:41.9897891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9898222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9898720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9899065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9899410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2lliand
2022-11-23T02:20:41.9899799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2lliand/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9900282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9900602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9901069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9901397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9901736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g2isx_h
2022-11-23T02:20:41.9902135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g2isx_h/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9902614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9902925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9903348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9903686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9904014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l_saguh
2022-11-23T02:20:41.9904406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l_saguh/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9904885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9905211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9905621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9905960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9906298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7qya0i6
2022-11-23T02:20:41.9906683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7qya0i6/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9907062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:41.9907406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:41.9907854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9908170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9908591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9908930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9909267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7zz5j75
2022-11-23T02:20:41.9909648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7zz5j75/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9910129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:41.9910454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:41.9910866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:41.9911201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:41.9911569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm2m6cuo
2022-11-23T02:20:41.9911960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm2m6cuo/_remote_module_non_scriptable.py
2022-11-23T02:20:41.9912316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:20:41.9912658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:41.9913002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:20:41.9913369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:41.9913814Z 2022-11-23 01:48:02,992 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-11-23T02:20:41.9914327Z 2022-11-23 01:48:02,992 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9914812Z 2022-11-23 01:48:03,002 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-11-23T02:20:41.9915304Z 2022-11-23 01:48:03,002 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9915778Z 2022-11-23 01:48:03,012 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-11-23T02:20:41.9916285Z 2022-11-23 01:48:03,012 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9916763Z 2022-11-23 01:48:03,032 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-11-23T02:20:41.9917198Z 2022-11-23 01:48:03,032 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-11-23T02:20:41.9917700Z 2022-11-23 01:48:03,032 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-11-23T02:20:41.9918181Z 2022-11-23 01:48:03,033 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-11-23T02:20:41.9918550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:41.9918907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:41.9919273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:41.9919636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T02:20:41.9920069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T02:20:41.9920626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:41.9921356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9922287Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9923150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9924099Z 2022-11-23 01:48:03,077 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-11-23T02:20:41.9925094Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9926071Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9927066Z 2022-11-23 01:48:03,087 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-11-23T02:20:41.9928042Z 2022-11-23 01:48:03,087 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers.
2022-11-23T02:20:41.9928789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-11-23T02:20:41.9929703Z 2022-11-23 01:48:03,087 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-11-23T02:20:41.9930687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:41.9931636Z 2022-11-23 01:48:03,088 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-11-23T02:20:41.9932585Z 2022-11-23 01:48:03,088 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-11-23T02:20:41.9933291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-11-23T02:20:41.9934133Z 2022-11-23 01:48:03,149 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-11-23T02:20:41.9935075Z 2022-11-23 01:48:03,150 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-11-23T02:20:41.9935734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:20:41.9936411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:20:41.9937077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-11-23T02:20:41.9937739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-11-23T02:20:41.9938679Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9939641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9940612Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9941576Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9942566Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9943546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-11-23T02:20:41.9944487Z 2022-11-23 01:48:03,269 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9945380Z 2022-11-23 01:48:03,269 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9946282Z 2022-11-23 01:48:03,269 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9947178Z 2022-11-23 01:48:03,269 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9948078Z 2022-11-23 01:48:03,278 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9948957Z 2022-11-23 01:48:03,278 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9949839Z 2022-11-23 01:48:03,286 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:41.9950842Z 2022-11-23 01:48:03,286 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-11-23T02:20:41.9952018Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9952817Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9953778Z 2022-11-23 01:48:03,317 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9955080Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9955886Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9956838Z 2022-11-23 01:48:03,317 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9957811Z 2022-11-23 01:48:03,317 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2
2022-11-23T02:20:41.9958366Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:41.9959388Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9960189Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9961161Z 2022-11-23 01:48:03,318 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9962143Z 2022-11-23 01:48:03,318 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2
2022-11-23T02:20:41.9962688Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:41.9963682Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:643: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-11-23T02:20:41.9964491Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-11-23T02:20:41.9965522Z 2022-11-23 01:48:03,318 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-11-23T02:20:41.9966586Z 2022-11-23 01:48:03,364 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:41.9967155Z         [ 1., -1.],
2022-11-23T02:20:41.9967448Z         [ 1.,  1.],
2022-11-23T02:20:41.9968179Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9968946Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:41.9969409Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9969778Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:41.9970139Z         [ 0.,  4.]])}
2022-11-23T02:20:41.9970904Z 2022-11-23 01:48:03,364 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9971442Z         [ 1.,  1.],
2022-11-23T02:20:41.9971785Z         [ 1., -1.],
2022-11-23T02:20:41.9972478Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9973243Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:41.9973722Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9974199Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:41.9974534Z         [ 0.,  4.]])}
2022-11-23T02:20:41.9975423Z 2022-11-23 01:48:03,364 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9976018Z         [-1., -1.],
2022-11-23T02:20:41.9976348Z         [-1.,  1.],
2022-11-23T02:20:41.9977066Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9977936Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:41.9978416Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9978790Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:41.9979136Z         [ 0.,  4.]])}
2022-11-23T02:20:41.9979929Z 2022-11-23 01:48:03,367 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9980504Z         [-1.,  1.],
2022-11-23T02:20:41.9980850Z         [-1., -1.],
2022-11-23T02:20:41.9981541Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9982303Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:41.9982763Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9983140Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:41.9983516Z         [ 0.,  4.]])}
2022-11-23T02:20:41.9984003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9984682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:20:41.9985648Z 2022-11-23 01:48:03,381 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:41.9986237Z         [-1.,  1.],
2022-11-23T02:20:41.9986575Z         [-1., -1.],
2022-11-23T02:20:41.9987266Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9988043Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9988517Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9989005Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:41.9989426Z         [-2.,  2.]])}
2022-11-23T02:20:41.9990213Z 2022-11-23 01:48:03,381 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:41.9990743Z         [ 1.,  1.],
2022-11-23T02:20:41.9991087Z         [ 1., -1.],
2022-11-23T02:20:41.9991791Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9992530Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:41.9993008Z tensor([[-1.,  1.],
2022-11-23T02:20:41.9993491Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:41.9993892Z         [-2.,  2.]])}
2022-11-23T02:20:41.9994593Z 2022-11-23 01:48:03,385 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2
2022-11-23T02:20:41.9995143Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:41.9995899Z 2022-11-23 01:48:03,385 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2
2022-11-23T02:20:41.9996432Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:41.9997353Z 2022-11-23 01:48:03,407 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:41.9997944Z         [-1., -1.],
2022-11-23T02:20:41.9998286Z         [-1.,  1.],
2022-11-23T02:20:41.9998996Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:41.9999774Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0000256Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0000611Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0001025Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0001813Z 2022-11-23 01:48:03,407 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:42.0002392Z         [ 1., -1.],
2022-11-23T02:20:42.0002685Z         [ 1.,  1.],
2022-11-23T02:20:42.0003417Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0004181Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0004652Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0005107Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0005451Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0006225Z 2022-11-23 01:48:03,407 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0006810Z         [-1.,  1.],
2022-11-23T02:20:42.0007149Z         [-1., -1.],
2022-11-23T02:20:42.0007833Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0008598Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0009093Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0009478Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0009825Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0010613Z 2022-11-23 01:48:03,407 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0011142Z         [ 1.,  1.],
2022-11-23T02:20:42.0011464Z         [ 1., -1.],
2022-11-23T02:20:42.0012152Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0012917Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0013399Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0013748Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0014098Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0014975Z 2022-11-23 01:48:03,425 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0015793Z         [-1.,  1.],
2022-11-23T02:20:42.0016021Z         [-1., -1.],
2022-11-23T02:20:42.0016429Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0016837Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:42.0017104Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0017371Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:42.0017607Z         [-2.,  2.]])}
2022-11-23T02:20:42.0018036Z 2022-11-23 01:48:03,426 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0018333Z         [ 1.,  1.],
2022-11-23T02:20:42.0018524Z         [ 1., -1.],
2022-11-23T02:20:42.0019003Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0019419Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:42.0019682Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0019936Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:42.0020167Z         [-2.,  2.]])}
2022-11-23T02:20:42.0020548Z 2022-11-23 01:48:03,429 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2
2022-11-23T02:20:42.0020894Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:42.0021288Z 2022-11-23 01:48:03,429 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2
2022-11-23T02:20:42.0021592Z                     to 1 to simulate uneven inputs.
2022-11-23T02:20:42.0022054Z 2022-11-23 01:48:03,449 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0022373Z         [-1.,  1.],
2022-11-23T02:20:42.0022551Z         [-1., -1.],
2022-11-23T02:20:42.0022935Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0023353Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0023605Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0023814Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0024021Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0024435Z 2022-11-23 01:48:03,449 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:42.0024755Z         [-1., -1.],
2022-11-23T02:20:42.0024944Z         [-1.,  1.],
2022-11-23T02:20:42.0025338Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0025742Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0026002Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0026214Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0026400Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0026820Z 2022-11-23 01:48:03,449 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0027112Z         [ 1.,  1.],
2022-11-23T02:20:42.0027291Z         [ 1., -1.],
2022-11-23T02:20:42.0027673Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0028081Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0028349Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0028546Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0028741Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0029173Z 2022-11-23 01:48:03,449 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:42.0029475Z         [ 1., -1.],
2022-11-23T02:20:42.0029633Z         [ 1.,  1.],
2022-11-23T02:20:42.0030020Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0030437Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-11-23T02:20:42.0030684Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0030893Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-11-23T02:20:42.0031090Z         [ 0.,  4.]])}
2022-11-23T02:20:42.0031550Z 2022-11-23 01:48:03,459 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0031872Z         [-1.,  1.],
2022-11-23T02:20:42.0032063Z         [-1., -1.],
2022-11-23T02:20:42.0032430Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0032846Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:42.0033112Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0033375Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:42.0033619Z         [-2.,  2.]])}
2022-11-23T02:20:42.0034048Z 2022-11-23 01:48:03,460 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0034336Z         [ 1.,  1.],
2022-11-23T02:20:42.0034515Z         [ 1., -1.],
2022-11-23T02:20:42.0034889Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0035294Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-11-23T02:20:42.0035555Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0035804Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-11-23T02:20:42.0036030Z         [-2.,  2.]])}
2022-11-23T02:20:42.0036394Z 2022-11-23 01:48:03,465 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-11-23T02:20:42.0036833Z 2022-11-23 01:48:03,465 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-11-23T02:20:42.0037287Z 2022-11-23 01:48:03,466 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-11-23T02:20:42.0037729Z 2022-11-23 01:48:03,467 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-11-23T02:20:42.0038170Z 2022-11-23 01:48:03,467 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-11-23T02:20:42.0038644Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpm2lliand'>
2022-11-23T02:20:42.0038985Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:20:42.0039420Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp2g2isx_h'>
2022-11-23T02:20:42.0039755Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:20:42.0040180Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpnm2m6cuo'>
2022-11-23T02:20:42.0040505Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:20:42.0040929Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpx7qya0i6'>
2022-11-23T02:20:42.0041242Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:20:42.0041447Z ok (3.887s)
2022-11-23T02:20:42.0041552Z 
2022-11-23T02:20:42.0041751Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0041993Z Ran 1 test in 3.887s
2022-11-23T02:20:42.0042093Z 
2022-11-23T02:20:42.0042151Z OK
2022-11-23T02:20:42.0042240Z 
2022-11-23T02:20:42.0042327Z Generating XML reports...
2022-11-23T02:20:42.0042845Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014800.xml
2022-11-23T02:20:42.0043422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0043759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0044187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0044571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0044979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcjaw0nf
2022-11-23T02:20:42.0045394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcjaw0nf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0045865Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0046072Z 
2022-11-23T02:20:42.0046148Z Running tests...
2022-11-23T02:20:42.0046435Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0046903Z   test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7500
2022-11-23T02:20:42.0047325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7501
2022-11-23T02:20:42.0047640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7502
2022-11-23T02:20:42.0047963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7503
2022-11-23T02:20:42.0048279Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 7504
2022-11-23T02:20:42.0048597Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 7505
2022-11-23T02:20:42.0049042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0049374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0049801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0050140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0050480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp86f7f8
2022-11-23T02:20:42.0050883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp86f7f8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0051373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0051686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0052110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0052459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0052805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0iftqu1
2022-11-23T02:20:42.0053198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0iftqu1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0053688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0054015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0054437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0054780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0055302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmd18u0d_
2022-11-23T02:20:42.0055881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmd18u0d_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0056397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0056793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0057346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0057759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0058190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6sqc_uy
2022-11-23T02:20:42.0058719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6sqc_uy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0059168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0059657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0060079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0060576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0061044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0061424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9gyyhu65
2022-11-23T02:20:42.0061913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9gyyhu65/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0062505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0062894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0063341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0063768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0064180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbxgkcb6
2022-11-23T02:20:42.0064627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbxgkcb6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0065064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0065468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0065907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:20:42.0066310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0066703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:20:42.0067520Z 2022-11-23 01:48:08,730 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-11-23T02:20:42.0068133Z 2022-11-23 01:48:08,731 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-11-23T02:20:42.0068640Z 2022-11-23 01:48:08,740 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-11-23T02:20:42.0069244Z 2022-11-23 01:48:08,741 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-11-23T02:20:42.0069793Z 2022-11-23 01:48:08,750 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-11-23T02:20:42.0070368Z 2022-11-23 01:48:08,751 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-11-23T02:20:42.0071086Z 2022-11-23 01:48:08,770 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-11-23T02:20:42.0071696Z 2022-11-23 01:48:08,770 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-11-23T02:20:42.0072254Z 2022-11-23 01:48:08,771 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-11-23T02:20:42.0072783Z 2022-11-23 01:48:08,771 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-11-23T02:20:42.0073215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0073677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0074135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0074576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0074959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-11-23T02:20:42.0075405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-11-23T02:20:42.0076026Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0076628Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0077281Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0077866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0078446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0079019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-11-23T02:20:42.0079580Z 2022-11-23 01:48:08,854 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-11-23T02:20:42.0080140Z 2022-11-23 01:48:08,854 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-11-23T02:20:42.0080734Z 2022-11-23 01:48:08,855 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-11-23T02:20:42.0081280Z 2022-11-23 01:48:08,855 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-11-23T02:20:42.0081769Z 2022-11-23 01:48:08,855 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-11-23T02:20:42.0082324Z 2022-11-23 01:48:08,861 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:42.0082944Z 2022-11-23 01:48:08,861 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-11-23T02:20:42.0083504Z 2022-11-23 01:48:08,878 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-11-23T02:20:42.0083967Z 2022-11-23 01:48:08,878 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-11-23T02:20:42.0084554Z 2022-11-23 01:48:08,906 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:42.0085017Z         [ 1., -1.],
2022-11-23T02:20:42.0085236Z         [ 1.,  1.],
2022-11-23T02:20:42.0085686Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0086175Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0086536Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0086822Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:42.0087115Z         [  0., -16.]])}
2022-11-23T02:20:42.0087552Z 2022-11-23 01:48:08,954 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:42.0088174Z 2022-11-23 01:48:08,954 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-11-23T02:20:42.0088786Z 2022-11-23 01:48:08,966 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:42.0089434Z 2022-11-23 01:48:08,967 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-11-23T02:20:42.0090069Z 2022-11-23 01:48:08,972 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:42.0090493Z         [-1., -1.],
2022-11-23T02:20:42.0090724Z         [-1.,  1.],
2022-11-23T02:20:42.0091172Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0091662Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0091950Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0092274Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:42.0092620Z         [ 16., -16.]])}
2022-11-23T02:20:42.0093026Z 2022-11-23 01:48:08,978 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-11-23T02:20:42.0093661Z 2022-11-23 01:48:08,979 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-11-23T02:20:42.0094289Z 2022-11-23 01:48:08,992 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0094659Z         [ 1.,  1.],
2022-11-23T02:20:42.0095004Z         [ 1., -1.],
2022-11-23T02:20:42.0095579Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0096367Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0096708Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0096935Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:42.0097230Z         [ 0., 16.]])}
2022-11-23T02:20:42.0097713Z 2022-11-23 01:48:09,001 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0098082Z         [-1.,  1.],
2022-11-23T02:20:42.0098334Z         [-1., -1.],
2022-11-23T02:20:42.0098773Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0099207Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0099531Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0099881Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:42.0100176Z         [-16.,  16.]])}
2022-11-23T02:20:42.0100661Z 2022-11-23 01:48:09,016 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:42.0101040Z         [ 1., -1.],
2022-11-23T02:20:42.0101297Z         [ 1.,  1.],
2022-11-23T02:20:42.0101701Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0102196Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0102526Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0102848Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:42.0103117Z         [  0., -16.]])}
2022-11-23T02:20:42.0103687Z 2022-11-23 01:48:09,017 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:42.0104098Z         [-1., -1.],
2022-11-23T02:20:42.0104307Z         [-1.,  1.],
2022-11-23T02:20:42.0104751Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0105265Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0105549Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0105875Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:42.0106208Z         [ 16., -16.]])}
2022-11-23T02:20:42.0106717Z 2022-11-23 01:48:09,017 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0107055Z         [-1.,  1.],
2022-11-23T02:20:42.0107333Z         [-1., -1.],
2022-11-23T02:20:42.0107772Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0108207Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0108562Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0108975Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:42.0109223Z         [-16.,  16.]])}
2022-11-23T02:20:42.0109718Z 2022-11-23 01:48:09,020 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0110066Z         [ 1.,  1.],
2022-11-23T02:20:42.0110338Z         [ 1., -1.],
2022-11-23T02:20:42.0110764Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0111234Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0111557Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0111798Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:42.0112074Z         [ 0., 16.]])}
2022-11-23T02:20:42.0112571Z 2022-11-23 01:48:09,036 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-11-23T02:20:42.0112943Z         [-1., -1.],
2022-11-23T02:20:42.0113146Z         [-1.,  1.],
2022-11-23T02:20:42.0113603Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0114119Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0114399Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0114811Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-11-23T02:20:42.0115126Z         [ 16., -16.]])}
2022-11-23T02:20:42.0115631Z 2022-11-23 01:48:09,039 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-11-23T02:20:42.0115763Z         [-1.,  1.],
2022-11-23T02:20:42.0115883Z         [-1., -1.],
2022-11-23T02:20:42.0116195Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0116441Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0116566Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0116761Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-11-23T02:20:42.0116888Z         [-16.,  16.]])}
2022-11-23T02:20:42.0117200Z 2022-11-23 01:48:09,039 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-11-23T02:20:42.0117353Z         [ 1., -1.],
2022-11-23T02:20:42.0117525Z         [ 1.,  1.],
2022-11-23T02:20:42.0117845Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0118098Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-11-23T02:20:42.0118223Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0118417Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-11-23T02:20:42.0118544Z         [  0., -16.]])}
2022-11-23T02:20:42.0118856Z 2022-11-23 01:48:09,039 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-11-23T02:20:42.0118989Z         [ 1.,  1.],
2022-11-23T02:20:42.0119129Z         [ 1., -1.],
2022-11-23T02:20:42.0119436Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-11-23T02:20:42.0119684Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-11-23T02:20:42.0119811Z tensor([[-1.,  1.],
2022-11-23T02:20:42.0119948Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-11-23T02:20:42.0120051Z         [ 0., 16.]])}
2022-11-23T02:20:42.0120304Z 2022-11-23 01:48:09,046 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-11-23T02:20:42.0120593Z 2022-11-23 01:48:09,047 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-11-23T02:20:42.0120907Z 2022-11-23 01:48:09,047 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-11-23T02:20:42.0121207Z 2022-11-23 01:48:09,048 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-11-23T02:20:42.0121488Z 2022-11-23 01:48:09,049 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-11-23T02:20:42.0121585Z ok (3.683s)
2022-11-23T02:20:42.0121593Z 
2022-11-23T02:20:42.0121821Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0121932Z Ran 1 test in 3.683s
2022-11-23T02:20:42.0121938Z 
2022-11-23T02:20:42.0121987Z OK
2022-11-23T02:20:42.0122034Z 
2022-11-23T02:20:42.0122106Z Generating XML reports...
2022-11-23T02:20:42.0122536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014805.xml
2022-11-23T02:20:42.0122842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0123016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0123327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0123493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0123719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwuya7t1
2022-11-23T02:20:42.0123945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwuya7t1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0124192Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0124237Z 
2022-11-23T02:20:42.0124297Z Running tests...
2022-11-23T02:20:42.0124521Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0124725Z   test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:42.0125071Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7890
2022-11-23T02:20:42.0125259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7891
2022-11-23T02:20:42.0125451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7892
2022-11-23T02:20:42.0125683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7893
2022-11-23T02:20:42.0125992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0126103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0126413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0126575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0126819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3zwm3vw
2022-11-23T02:20:42.0183309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3zwm3vw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0183775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0183907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0184191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0184319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0184498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar1r_qn_
2022-11-23T02:20:42.0184683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar1r_qn_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0184952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0185070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0185344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0185472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0185649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey0yr38q
2022-11-23T02:20:42.0185836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey0yr38q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0186098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0186212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0186478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0186605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0186779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a9xa5ft
2022-11-23T02:20:42.0186962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a9xa5ft/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0187119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0187275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0187427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0187579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0187745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0187918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0188085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0188379Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0188670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0188962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0189249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0189533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0189655Z ok (2.676s)
2022-11-23T02:20:42.0189663Z 
2022-11-23T02:20:42.0189854Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0189918Z Ran 1 test in 2.677s
2022-11-23T02:20:42.0189923Z 
2022-11-23T02:20:42.0189975Z OK
2022-11-23T02:20:42.0189980Z 
2022-11-23T02:20:42.0190054Z Generating XML reports...
2022-11-23T02:20:42.0190423Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014811.xml
2022-11-23T02:20:42.0190688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0190802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0191078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0191205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0191379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7asfsitb
2022-11-23T02:20:42.0191569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7asfsitb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0191817Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0191822Z 
2022-11-23T02:20:42.0191884Z Running tests...
2022-11-23T02:20:42.0192075Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0192306Z   test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8205
2022-11-23T02:20:42.0192452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8206
2022-11-23T02:20:42.0192598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8207
2022-11-23T02:20:42.0192741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8208
2022-11-23T02:20:42.0193006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0193121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0193398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0193527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0193703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwuyhng2
2022-11-23T02:20:42.0193893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwuyhng2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0194157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0194272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0194542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0194672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0194848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahzylaly
2022-11-23T02:20:42.0195069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahzylaly/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0195332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0195447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0195720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0195846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0196024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp315srp1t
2022-11-23T02:20:42.0196239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp315srp1t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0196504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0196618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0196894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0197020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0197196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6furu9ne
2022-11-23T02:20:42.0197382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6furu9ne/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0197538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0197692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0197846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0198000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0198059Z ok (2.979s)
2022-11-23T02:20:42.0198067Z 
2022-11-23T02:20:42.0198256Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0198323Z Ran 1 test in 2.979s
2022-11-23T02:20:42.0198328Z 
2022-11-23T02:20:42.0198380Z OK
2022-11-23T02:20:42.0198385Z 
2022-11-23T02:20:42.0198461Z Generating XML reports...
2022-11-23T02:20:42.0198824Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014816.xml
2022-11-23T02:20:42.0199087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0199205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0199480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0199609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0199790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy02jhj4
2022-11-23T02:20:42.0199977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy02jhj4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0200223Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0200228Z 
2022-11-23T02:20:42.0200292Z Running tests...
2022-11-23T02:20:42.0200475Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0200717Z   test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8428
2022-11-23T02:20:42.0200867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8429
2022-11-23T02:20:42.0201014Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8430
2022-11-23T02:20:42.0201157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8431
2022-11-23T02:20:42.0201451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0201570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0201843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0201971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0202141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y7xegig
2022-11-23T02:20:42.0202357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y7xegig/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0202624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0202746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0203028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0203161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0203348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1po00s5
2022-11-23T02:20:42.0203546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1po00s5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0203816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0203931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0204211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0204348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0204535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrgsln39
2022-11-23T02:20:42.0204730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrgsln39/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0205085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0205212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0205496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0205617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0205805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiadhtxvt
2022-11-23T02:20:42.0206003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiadhtxvt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0206167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0206335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0206495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0206655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0206722Z ok (2.474s)
2022-11-23T02:20:42.0206728Z 
2022-11-23T02:20:42.0206925Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0206988Z Ran 1 test in 2.475s
2022-11-23T02:20:42.0206993Z 
2022-11-23T02:20:42.0207057Z OK
2022-11-23T02:20:42.0207062Z 
2022-11-23T02:20:42.0207146Z Generating XML reports...
2022-11-23T02:20:42.0207516Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014820.xml
2022-11-23T02:20:42.0207791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0207947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0208229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0208366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0208539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6d8tfki
2022-11-23T02:20:42.0208738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6d8tfki/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0209023Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0209028Z 
2022-11-23T02:20:42.0209100Z Running tests...
2022-11-23T02:20:42.0209298Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0209553Z   test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8671
2022-11-23T02:20:42.0209709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8672
2022-11-23T02:20:42.0209861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8673
2022-11-23T02:20:42.0210013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8674
2022-11-23T02:20:42.0210273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0210395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0210681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0210815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0211000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxfx1x33
2022-11-23T02:20:42.0211197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxfx1x33/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0211471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0211593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0211860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0211997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0212183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmenang2d
2022-11-23T02:20:42.0212378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmenang2d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0212653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0212777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0213053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0213188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0213372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperfq6dxx
2022-11-23T02:20:42.0213555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperfq6dxx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0213826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0213951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0214230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0214392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0214577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6fs43xx
2022-11-23T02:20:42.0214774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6fs43xx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0215063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0215227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0215375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0215585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0215653Z ok (2.575s)
2022-11-23T02:20:42.0215658Z 
2022-11-23T02:20:42.0215858Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0215933Z Ran 1 test in 2.575s
2022-11-23T02:20:42.0215938Z 
2022-11-23T02:20:42.0216001Z OK
2022-11-23T02:20:42.0216006Z 
2022-11-23T02:20:42.0216089Z Generating XML reports...
2022-11-23T02:20:42.0216460Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014825.xml
2022-11-23T02:20:42.0216722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0216846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0217127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0217264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0217450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt2z7tqt
2022-11-23T02:20:42.0217646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt2z7tqt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0217903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0217909Z 
2022-11-23T02:20:42.0217983Z Running tests...
2022-11-23T02:20:42.0218181Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0218417Z   test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8914
2022-11-23T02:20:42.0218574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8915
2022-11-23T02:20:42.0218729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8916
2022-11-23T02:20:42.0218881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8917
2022-11-23T02:20:42.0219156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0219280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0219560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0219698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0219871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplt5y_cok
2022-11-23T02:20:42.0220069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplt5y_cok/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0220341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0220466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0220748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0220884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0221106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4p4qc6f
2022-11-23T02:20:42.0221305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4p4qc6f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0221578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0221689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0221970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0222166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0222352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcblf1e5k
2022-11-23T02:20:42.0222549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcblf1e5k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0222826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0222948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0223231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0223352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0223537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp748id4xg
2022-11-23T02:20:42.0223730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp748id4xg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0223897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0224058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0224222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0224381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0224447Z ok (2.474s)
2022-11-23T02:20:42.0224452Z 
2022-11-23T02:20:42.0224650Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0224712Z Ran 1 test in 2.474s
2022-11-23T02:20:42.0224717Z 
2022-11-23T02:20:42.0224778Z OK
2022-11-23T02:20:42.0224783Z 
2022-11-23T02:20:42.0224867Z Generating XML reports...
2022-11-23T02:20:42.0225236Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014829.xml
2022-11-23T02:20:42.0225511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0225633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0225915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0226051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0226221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4o1zme9l
2022-11-23T02:20:42.0226416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4o1zme9l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0226670Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0226675Z 
2022-11-23T02:20:42.0226746Z Running tests...
2022-11-23T02:20:42.0226943Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0227191Z   test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9157
2022-11-23T02:20:42.0227346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9158
2022-11-23T02:20:42.0227534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9159
2022-11-23T02:20:42.0227686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9160
2022-11-23T02:20:42.0227948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0228072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0228353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0228486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0228705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1cch9zz
2022-11-23T02:20:42.0228901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1cch9zz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0229174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0229299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0229579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0229701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0229886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv21djzyf
2022-11-23T02:20:42.0230080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv21djzyf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0230353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0230476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0230755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0230894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0231081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafiiza9i
2022-11-23T02:20:42.0231264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafiiza9i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0231546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0231666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0231948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0232082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0232267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4penfdg0
2022-11-23T02:20:42.0232463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4penfdg0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0232629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0232787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0232934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0233096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0233165Z ok (2.477s)
2022-11-23T02:20:42.0233172Z 
2022-11-23T02:20:42.0233370Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0233444Z Ran 1 test in 2.477s
2022-11-23T02:20:42.0233449Z 
2022-11-23T02:20:42.0233508Z OK
2022-11-23T02:20:42.0233513Z 
2022-11-23T02:20:42.0233595Z Generating XML reports...
2022-11-23T02:20:42.0233995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014834.xml
2022-11-23T02:20:42.0234255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0234381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0234662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0234798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0234985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgppad96a
2022-11-23T02:20:42.0235228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgppad96a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0235485Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0235491Z 
2022-11-23T02:20:42.0235564Z Running tests...
2022-11-23T02:20:42.0235761Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0236003Z   test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9400
2022-11-23T02:20:42.0236158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9401
2022-11-23T02:20:42.0236309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9402
2022-11-23T02:20:42.0236461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9403
2022-11-23T02:20:42.0236738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0236862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0237144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0237281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0237452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dr7jwse
2022-11-23T02:20:42.0237649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dr7jwse/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0237921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0238042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0238320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0238456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0238641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex9qoraf
2022-11-23T02:20:42.0238838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex9qoraf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0239109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0239218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0239499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0239632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0239819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphv3l79v2
2022-11-23T02:20:42.0240016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphv3l79v2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0240285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0240408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0240714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0240853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0241024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_cp_5c4
2022-11-23T02:20:42.0241216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_cp_5c4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0241379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0241573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0241731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0241966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0242067Z ok (2.676s)
2022-11-23T02:20:42.0242072Z 
2022-11-23T02:20:42.0242275Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0242336Z Ran 1 test in 2.676s
2022-11-23T02:20:42.0242342Z 
2022-11-23T02:20:42.0242402Z OK
2022-11-23T02:20:42.0242407Z 
2022-11-23T02:20:42.0242491Z Generating XML reports...
2022-11-23T02:20:42.0242860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014838.xml
2022-11-23T02:20:42.0243138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0243263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0243543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0243681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0243866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz_4wjce
2022-11-23T02:20:42.0244048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz_4wjce/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0244303Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0244308Z 
2022-11-23T02:20:42.0244380Z Running tests...
2022-11-23T02:20:42.0244577Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0244822Z   test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9643
2022-11-23T02:20:42.0245054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9644
2022-11-23T02:20:42.0245208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9645
2022-11-23T02:20:42.0245360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9646
2022-11-23T02:20:42.0245628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0245755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0246038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0246174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0246358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgihgx0e9
2022-11-23T02:20:42.0246553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgihgx0e9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0246829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0246954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0247272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0247395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0247579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8__27zbj
2022-11-23T02:20:42.0247769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8__27zbj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0248042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0248165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0248474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0248607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0248791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2shf6i2
2022-11-23T02:20:42.0248974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2shf6i2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0249247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0249370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0249651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0249784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0249972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2bz14wh
2022-11-23T02:20:42.0250166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2bz14wh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0250330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0250496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0250645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0250808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0250874Z ok (2.577s)
2022-11-23T02:20:42.0250879Z 
2022-11-23T02:20:42.0251074Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0251149Z Ran 1 test in 2.577s
2022-11-23T02:20:42.0251154Z 
2022-11-23T02:20:42.0251214Z OK
2022-11-23T02:20:42.0251221Z 
2022-11-23T02:20:42.0251304Z Generating XML reports...
2022-11-23T02:20:42.0251725Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014842.xml
2022-11-23T02:20:42.0252110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0252241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0252746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0253002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0253263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h1wa9gv
2022-11-23T02:20:42.0253465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h1wa9gv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0253720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0253728Z 
2022-11-23T02:20:42.0253801Z Running tests...
2022-11-23T02:20:42.0254012Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0254798Z   test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.259s)
2022-11-23T02:20:42.0254805Z 
2022-11-23T02:20:42.0255210Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0255273Z Ran 1 test in 0.259s
2022-11-23T02:20:42.0255292Z 
2022-11-23T02:20:42.0255351Z OK (skipped=1)
2022-11-23T02:20:42.0255357Z 
2022-11-23T02:20:42.0255441Z Generating XML reports...
2022-11-23T02:20:42.0255867Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014847.xml
2022-11-23T02:20:42.0256141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0256265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0256547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0256684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0256873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhqboptm
2022-11-23T02:20:42.0257060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhqboptm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0257314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0257321Z 
2022-11-23T02:20:42.0257394Z Running tests...
2022-11-23T02:20:42.0257589Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0257833Z   test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9880
2022-11-23T02:20:42.0257992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9881
2022-11-23T02:20:42.0258146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9882
2022-11-23T02:20:42.0258296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9883
2022-11-23T02:20:42.0258573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0258685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0258969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0259107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0259295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3dgydc8
2022-11-23T02:20:42.0259493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3dgydc8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0259771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0259897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0260180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0260302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0260492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjojbn90
2022-11-23T02:20:42.0260688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjojbn90/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0260961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0261083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0261401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0261539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0261723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuagd97ym
2022-11-23T02:20:42.0261920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuagd97ym/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0262183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0262334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0262617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0262752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0262939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsstqtuyu
2022-11-23T02:20:42.0263135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsstqtuyu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0263299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0263464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0263610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0263774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0263844Z ok (2.582s)
2022-11-23T02:20:42.0263849Z 
2022-11-23T02:20:42.0264046Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0264120Z Ran 1 test in 2.583s
2022-11-23T02:20:42.0264125Z 
2022-11-23T02:20:42.0264185Z OK
2022-11-23T02:20:42.0264189Z 
2022-11-23T02:20:42.0264273Z Generating XML reports...
2022-11-23T02:20:42.0264645Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014849.xml
2022-11-23T02:20:42.0264923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0265033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0265314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0265448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0265634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc85yfky
2022-11-23T02:20:42.0265827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc85yfky/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0266085Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0266091Z 
2022-11-23T02:20:42.0266168Z Running tests...
2022-11-23T02:20:42.0266360Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0267071Z   test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.252s)
2022-11-23T02:20:42.0267079Z 
2022-11-23T02:20:42.0267275Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0267338Z Ran 1 test in 0.253s
2022-11-23T02:20:42.0267343Z 
2022-11-23T02:20:42.0267414Z OK (skipped=1)
2022-11-23T02:20:42.0267419Z 
2022-11-23T02:20:42.0267503Z Generating XML reports...
2022-11-23T02:20:42.0267897Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014853.xml
2022-11-23T02:20:42.0268174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0268298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0268577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0268711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0268882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjk7xa4p
2022-11-23T02:20:42.0269110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjk7xa4p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0269365Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0269370Z 
2022-11-23T02:20:42.0269441Z Running tests...
2022-11-23T02:20:42.0269637Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0269886Z   test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10133
2022-11-23T02:20:42.0270043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10134
2022-11-23T02:20:42.0270198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10135
2022-11-23T02:20:42.0270353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10136
2022-11-23T02:20:42.0270618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0270743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0271024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0271159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0271343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2za9ucrc
2022-11-23T02:20:42.0271537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2za9ucrc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0271811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0271933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0272213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0272337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0272522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc419jyrh
2022-11-23T02:20:42.0272718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc419jyrh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0272987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0273109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0273386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0273518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0273701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpens2xk2c
2022-11-23T02:20:42.0273883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpens2xk2c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0274154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0274276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0274586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0274721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0274903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4me3d2lc
2022-11-23T02:20:42.0275097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4me3d2lc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0275261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0275455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0275605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0275766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0275833Z ok (2.672s)
2022-11-23T02:20:42.0275838Z 
2022-11-23T02:20:42.0276039Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0276114Z Ran 1 test in 2.673s
2022-11-23T02:20:42.0276118Z 
2022-11-23T02:20:42.0276179Z OK
2022-11-23T02:20:42.0276184Z 
2022-11-23T02:20:42.0276267Z Generating XML reports...
2022-11-23T02:20:42.0276639Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014855.xml
2022-11-23T02:20:42.0276900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0277028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0277308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0277443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0277627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v6vwr2g
2022-11-23T02:20:42.0277824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v6vwr2g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0278079Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0278084Z 
2022-11-23T02:20:42.0278155Z Running tests...
2022-11-23T02:20:42.0278348Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0278579Z   test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10376
2022-11-23T02:20:42.0278739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10377
2022-11-23T02:20:42.0278895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10378
2022-11-23T02:20:42.0279048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10379
2022-11-23T02:20:42.0279323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0279446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0279731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0279870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0280040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vxot61j
2022-11-23T02:20:42.0280241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vxot61j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0280512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0280635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0280947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0281078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0281262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparpb2jux
2022-11-23T02:20:42.0281457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparpb2jux/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0281726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0281878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0282154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0282286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0282474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswyupa7v
2022-11-23T02:20:42.0282667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswyupa7v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0282933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0283057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0283338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0283460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0283649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1atm06j
2022-11-23T02:20:42.0283844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1atm06j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0284007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0284171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0284334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0284495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0284673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0284849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0285093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0285268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0285570Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0285873Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0286170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0286463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0286732Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0287001Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0287305Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0287594Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #4 to worker0, as the agent is shutting down
2022-11-23T02:20:42.0287859Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0288145Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2022-11-23T02:20:42.0288396Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0288661Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #3 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0288922Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0289213Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #5 to worker1: ECONNRESET: connection reset by peer (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2022-11-23T02:20:42.0289474Z [W tensorpipe_agent.cpp:915] RPC agent for worker2 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0289701Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0289963Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0290219Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0290475Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0290734Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0290953Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0291216Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0291477Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0291725Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0291988Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0292275Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0292535Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0292603Z ok (2.575s)
2022-11-23T02:20:42.0292609Z 
2022-11-23T02:20:42.0292808Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0292884Z Ran 1 test in 2.575s
2022-11-23T02:20:42.0292890Z 
2022-11-23T02:20:42.0292950Z OK
2022-11-23T02:20:42.0292955Z 
2022-11-23T02:20:42.0293038Z Generating XML reports...
2022-11-23T02:20:42.0293441Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014900.xml
2022-11-23T02:20:42.0293709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0293835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0294120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0294258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0294445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2sopj_g
2022-11-23T02:20:42.0294648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2sopj_g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0295008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0295016Z 
2022-11-23T02:20:42.0295091Z Running tests...
2022-11-23T02:20:42.0295286Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0295527Z   test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10621
2022-11-23T02:20:42.0295690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10622
2022-11-23T02:20:42.0295848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10623
2022-11-23T02:20:42.0296002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10624
2022-11-23T02:20:42.0296277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0296402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0296691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0296830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0297004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl25yzfbh
2022-11-23T02:20:42.0297207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl25yzfbh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0297478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0297603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0297881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0298016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0298204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmxhbw6z
2022-11-23T02:20:42.0298402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmxhbw6z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0298676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0298785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0299122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0299260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0299445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5_njv04
2022-11-23T02:20:42.0299634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5_njv04/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0299910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0300073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0300353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0300472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0300657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gjtejss
2022-11-23T02:20:42.0300851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gjtejss/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0301015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0301178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0301340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0301502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0301682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0301859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0302019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0302196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0302501Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0302802Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0303103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0303403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0303674Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0303948Z [W tensorpipe_agent.cpp:678] RPC agent for worker3 encountered error when sending response to request #4 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0304210Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0304471Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0304739Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0304946Z [E container.cpp:256] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0305274Z [W tensorpipe_agent.cpp:915] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2022-11-23T02:20:42.0305542Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #9 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0305803Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0306095Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #11 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0306363Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0306625Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0306889Z [W tensorpipe_agent.cpp:939] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0307147Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0307404Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0307626Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0307886Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0308147Z [W tensorpipe_agent.cpp:915] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0308408Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0308466Z ok (2.676s)
2022-11-23T02:20:42.0308471Z 
2022-11-23T02:20:42.0308672Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0308748Z Ran 1 test in 2.677s
2022-11-23T02:20:42.0308755Z 
2022-11-23T02:20:42.0308816Z OK
2022-11-23T02:20:42.0308821Z 
2022-11-23T02:20:42.0308905Z Generating XML reports...
2022-11-23T02:20:42.0309278Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014905.xml
2022-11-23T02:20:42.0309555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0309679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0309964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0310089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0310279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqh50uhc
2022-11-23T02:20:42.0310508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqh50uhc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0310766Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0310771Z 
2022-11-23T02:20:42.0310844Z Running tests...
2022-11-23T02:20:42.0311039Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0311286Z   test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10866
2022-11-23T02:20:42.0311445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10867
2022-11-23T02:20:42.0311622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10868
2022-11-23T02:20:42.0311777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10869
2022-11-23T02:20:42.0312055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0312182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0312466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0312600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0312785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtlqil5o
2022-11-23T02:20:42.0312983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtlqil5o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0313260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0313372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0313655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0313793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0313977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k254vz3
2022-11-23T02:20:42.0314174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k254vz3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0314448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0314571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0314852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0314975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0315160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3c7b35c
2022-11-23T02:20:42.0315355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3c7b35c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0315628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0315751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0316032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0316168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0316354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhdjllmp
2022-11-23T02:20:42.0316555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhdjllmp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0316705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0316871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0317058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0317222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0317293Z ok (2.684s)
2022-11-23T02:20:42.0317298Z 
2022-11-23T02:20:42.0317499Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0317577Z Ran 1 test in 2.684s
2022-11-23T02:20:42.0317582Z 
2022-11-23T02:20:42.0317642Z OK
2022-11-23T02:20:42.0317647Z 
2022-11-23T02:20:42.0317717Z Generating XML reports...
2022-11-23T02:20:42.0318087Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014909.xml
2022-11-23T02:20:42.0318387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0318512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0318795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0318931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0319118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm76uddin
2022-11-23T02:20:42.0319314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm76uddin/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0319571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0319577Z 
2022-11-23T02:20:42.0319638Z Running tests...
2022-11-23T02:20:42.0319834Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0320069Z   test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11109
2022-11-23T02:20:42.0320223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11110
2022-11-23T02:20:42.0320380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11111
2022-11-23T02:20:42.0320530Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11112
2022-11-23T02:20:42.0320811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0320934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0321221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0321347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0321535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8i7qiia
2022-11-23T02:20:42.0321809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0321932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0322216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0322349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0322536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xnm4z5b
2022-11-23T02:20:42.0322731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8i7qiia/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0322908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xnm4z5b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0323183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0323307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0323653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0323789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0323972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi978z15j
2022-11-23T02:20:42.0324167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi978z15j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0324441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0324564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0324871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0325097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0325286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e1ggvw1
2022-11-23T02:20:42.0325487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e1ggvw1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0325653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0325817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0325979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0326140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0326193Z ok (2.574s)
2022-11-23T02:20:42.0326214Z 
2022-11-23T02:20:42.0326399Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0326474Z Ran 1 test in 2.574s
2022-11-23T02:20:42.0326479Z 
2022-11-23T02:20:42.0326540Z OK
2022-11-23T02:20:42.0326545Z 
2022-11-23T02:20:42.0326630Z Generating XML reports...
2022-11-23T02:20:42.0327004Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014914.xml
2022-11-23T02:20:42.0327280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0327403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0327684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0327806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0327990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26i32l0y
2022-11-23T02:20:42.0328188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26i32l0y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0328442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0328446Z 
2022-11-23T02:20:42.0328522Z Running tests...
2022-11-23T02:20:42.0328716Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0328958Z   test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11352
2022-11-23T02:20:42.0329115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11353
2022-11-23T02:20:42.0329256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11354
2022-11-23T02:20:42.0329408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11355
2022-11-23T02:20:42.0329687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0329811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0330096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0330266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0330453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5867gzpr
2022-11-23T02:20:42.0330647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5867gzpr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0330919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0331028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0331310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0331471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0331653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt316lmz5
2022-11-23T02:20:42.0331845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt316lmz5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0332118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0332241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0332522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0332657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0332828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1iuj7ib
2022-11-23T02:20:42.0333095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0333214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0333489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0333624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0333809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp154mhm67
2022-11-23T02:20:42.0334003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1iuj7ib/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0334185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp154mhm67/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0334337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0334500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0334662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0334823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0334992Z ok (2.575s)
2022-11-23T02:20:42.0334998Z 
2022-11-23T02:20:42.0335200Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0335276Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0335281Z 
2022-11-23T02:20:42.0335341Z OK
2022-11-23T02:20:42.0335346Z 
2022-11-23T02:20:42.0335416Z Generating XML reports...
2022-11-23T02:20:42.0335791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014918.xml
2022-11-23T02:20:42.0336067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0336192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0336476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0336612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0336848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh_jnn9n
2022-11-23T02:20:42.0337045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh_jnn9n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0337297Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0337302Z 
2022-11-23T02:20:42.0337362Z Running tests...
2022-11-23T02:20:42.0337563Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0337806Z   test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11595
2022-11-23T02:20:42.0338002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11596
2022-11-23T02:20:42.0338157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11597
2022-11-23T02:20:42.0338311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11598
2022-11-23T02:20:42.0338593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0338716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0339005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0339128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0339317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjiz65451
2022-11-23T02:20:42.0339512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjiz65451/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0339784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0339906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0340190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0340326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0340510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsqbfs9l
2022-11-23T02:20:42.0340694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsqbfs9l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0340968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0341094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0341375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0341511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0341702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp8od2go
2022-11-23T02:20:42.0341901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp8od2go/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0342174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0342301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0342570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0342704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0342889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplid_8q6f
2022-11-23T02:20:42.0343081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplid_8q6f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0343246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0343445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0343605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0343765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0343820Z ok (2.475s)
2022-11-23T02:20:42.0343838Z 
2022-11-23T02:20:42.0344052Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0344127Z Ran 1 test in 2.475s
2022-11-23T02:20:42.0344133Z 
2022-11-23T02:20:42.0344221Z OK
2022-11-23T02:20:42.0344226Z 
2022-11-23T02:20:42.0344310Z Generating XML reports...
2022-11-23T02:20:42.0344684Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014922.xml
2022-11-23T02:20:42.0344956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0345082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0345365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0345489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0345671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_63zscbu
2022-11-23T02:20:42.0345867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_63zscbu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0346125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0346130Z 
2022-11-23T02:20:42.0346201Z Running tests...
2022-11-23T02:20:42.0346398Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0346640Z   test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11838
2022-11-23T02:20:42.0346798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11839
2022-11-23T02:20:42.0346952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11840
2022-11-23T02:20:42.0347094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11841
2022-11-23T02:20:42.0347371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0347497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0347785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0347920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0348103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6mh65vo
2022-11-23T02:20:42.0348297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6mh65vo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0348567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0348677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0348960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0349093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0349281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlltrr_v
2022-11-23T02:20:42.0349478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlltrr_v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0349748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0349901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0350184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0350317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0350491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqputiy_
2022-11-23T02:20:42.0350685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqputiy_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0350956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0351112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0351392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0351525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0351714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_j4iip4
2022-11-23T02:20:42.0351906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_j4iip4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0352056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0352220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0352381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0352543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0352611Z ok (2.676s)
2022-11-23T02:20:42.0352616Z 
2022-11-23T02:20:42.0352814Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0352889Z Ran 1 test in 2.676s
2022-11-23T02:20:42.0352894Z 
2022-11-23T02:20:42.0352954Z OK
2022-11-23T02:20:42.0352961Z 
2022-11-23T02:20:42.0353046Z Generating XML reports...
2022-11-23T02:20:42.0353408Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014927.xml
2022-11-23T02:20:42.0353683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0353807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0354086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0354223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0354408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bdb1f3h
2022-11-23T02:20:42.0354602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bdb1f3h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0354862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0354867Z 
2022-11-23T02:20:42.0354928Z Running tests...
2022-11-23T02:20:42.0355125Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0355373Z   test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12081
2022-11-23T02:20:42.0355532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12082
2022-11-23T02:20:42.0355685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12083
2022-11-23T02:20:42.0355841Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12084
2022-11-23T02:20:42.0356119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0356242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0356556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0356680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0356868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1vaxo7g
2022-11-23T02:20:42.0357063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1vaxo7g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0357337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0357495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0357775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0357910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0358096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7innbjf
2022-11-23T02:20:42.0358278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7innbjf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0358656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0358779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0359061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0359199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0359384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxye4mib4
2022-11-23T02:20:42.0359579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxye4mib4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0359856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0359979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0360246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0360383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0360567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt935vyyn
2022-11-23T02:20:42.0360759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt935vyyn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0360925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0361086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0361249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0361412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0361467Z ok (2.577s)
2022-11-23T02:20:42.0361488Z 
2022-11-23T02:20:42.0361674Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0361749Z Ran 1 test in 2.577s
2022-11-23T02:20:42.0361754Z 
2022-11-23T02:20:42.0361814Z OK
2022-11-23T02:20:42.0361819Z 
2022-11-23T02:20:42.0361901Z Generating XML reports...
2022-11-23T02:20:42.0362272Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014931.xml
2022-11-23T02:20:42.0362551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0362676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0363003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0363126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0363312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1n_3v3k
2022-11-23T02:20:42.0363505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1n_3v3k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0363760Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0363765Z 
2022-11-23T02:20:42.0363838Z Running tests...
2022-11-23T02:20:42.0364034Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0364318Z   test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12324
2022-11-23T02:20:42.0364475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12325
2022-11-23T02:20:42.0364632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12326
2022-11-23T02:20:42.0364772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12327
2022-11-23T02:20:42.0365118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0365242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0365527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0365661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0365848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi5c542v
2022-11-23T02:20:42.0366044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi5c542v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0366318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0366428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0366712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0366848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0367031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61_w3490
2022-11-23T02:20:42.0367223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61_w3490/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0367495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0367619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0367902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0368038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0368210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1cunnn9
2022-11-23T02:20:42.0368407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1cunnn9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0368684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0368807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0369086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0369223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0369407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7n0517i
2022-11-23T02:20:42.0369636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7n0517i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0369789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0369956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0370118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0370276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0370344Z ok (2.679s)
2022-11-23T02:20:42.0370349Z 
2022-11-23T02:20:42.0370595Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0370672Z Ran 1 test in 2.679s
2022-11-23T02:20:42.0370676Z 
2022-11-23T02:20:42.0370741Z OK
2022-11-23T02:20:42.0370745Z 
2022-11-23T02:20:42.0370833Z Generating XML reports...
2022-11-23T02:20:42.0371196Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014936.xml
2022-11-23T02:20:42.0371470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0371592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0371874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0372010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0372195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nd5bryb
2022-11-23T02:20:42.0372395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nd5bryb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0372653Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0372658Z 
2022-11-23T02:20:42.0372730Z Running tests...
2022-11-23T02:20:42.0372915Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0373159Z   test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12567
2022-11-23T02:20:42.0373317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12568
2022-11-23T02:20:42.0373474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12569
2022-11-23T02:20:42.0373633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12570
2022-11-23T02:20:42.0373913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0374037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0374321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0374445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0374632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu44ibq8v
2022-11-23T02:20:42.0374828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu44ibq8v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0375209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0375336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0375621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0375758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0375944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm8uua21
2022-11-23T02:20:42.0376139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm8uua21/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0376455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0376578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0376861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0376995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0377177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzz_k782
2022-11-23T02:20:42.0377408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzz_k782/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0377682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0377804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0378074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0378209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0378393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr05rnk2_
2022-11-23T02:20:42.0378585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr05rnk2_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0378750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0378916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0379081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0379245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0379313Z ok (2.576s)
2022-11-23T02:20:42.0379318Z 
2022-11-23T02:20:42.0379505Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0379580Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0379585Z 
2022-11-23T02:20:42.0379645Z OK
2022-11-23T02:20:42.0379649Z 
2022-11-23T02:20:42.0379733Z Generating XML reports...
2022-11-23T02:20:42.0380102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014940.xml
2022-11-23T02:20:42.0380377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0380502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0380784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0380905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0381091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__w2u853
2022-11-23T02:20:42.0381283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__w2u853/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0381540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0381545Z 
2022-11-23T02:20:42.0381617Z Running tests...
2022-11-23T02:20:42.0381813Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0382067Z   test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12810
2022-11-23T02:20:42.0382225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12811
2022-11-23T02:20:42.0382379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12812
2022-11-23T02:20:42.0382519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12813
2022-11-23T02:20:42.0382829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0382951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0383232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0383367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0383553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2gu7h76
2022-11-23T02:20:42.0383747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2gu7h76/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0384053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0384163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0384447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0384581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0384763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwc4k1zj
2022-11-23T02:20:42.0384960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwc4k1zj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0385232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0385357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0385640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0385777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0385947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihdwurmw
2022-11-23T02:20:42.0386146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihdwurmw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0386418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0386540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0386820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0386953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0387136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4wzisxs
2022-11-23T02:20:42.0387328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4wzisxs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0387495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0387649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0387809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0387969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0388037Z ok (12.491s)
2022-11-23T02:20:42.0388042Z 
2022-11-23T02:20:42.0388239Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0388315Z Ran 1 test in 12.491s
2022-11-23T02:20:42.0388320Z 
2022-11-23T02:20:42.0388380Z OK
2022-11-23T02:20:42.0388387Z 
2022-11-23T02:20:42.0388470Z Generating XML reports...
2022-11-23T02:20:42.0388829Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014945.xml
2022-11-23T02:20:42.0389103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0389258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0389541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0389675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0389860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgm4o6hx
2022-11-23T02:20:42.0390056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgm4o6hx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0390311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0390345Z 
2022-11-23T02:20:42.0390418Z Running tests...
2022-11-23T02:20:42.0390601Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0391312Z   test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.251s)
2022-11-23T02:20:42.0391318Z 
2022-11-23T02:20:42.0391516Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0391591Z Ran 1 test in 0.252s
2022-11-23T02:20:42.0391596Z 
2022-11-23T02:20:42.0391668Z OK (skipped=1)
2022-11-23T02:20:42.0391672Z 
2022-11-23T02:20:42.0391755Z Generating XML reports...
2022-11-23T02:20:42.0392126Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014959.xml
2022-11-23T02:20:42.0392399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0392527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0392811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0392933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0393117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wp2qi8r
2022-11-23T02:20:42.0393312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wp2qi8r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0393567Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0393574Z 
2022-11-23T02:20:42.0393645Z Running tests...
2022-11-23T02:20:42.0393841Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0394083Z   test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13067
2022-11-23T02:20:42.0394244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13068
2022-11-23T02:20:42.0394387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13069
2022-11-23T02:20:42.0394541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13070
2022-11-23T02:20:42.0394816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0394940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0395223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0395361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0395546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzyp53ku
2022-11-23T02:20:42.0395743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzyp53ku/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0396050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0396162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0396445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0396582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0396765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88ekvn6k
2022-11-23T02:20:42.0397000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88ekvn6k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0397270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0397392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0397675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0397795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0397980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj5wqoch
2022-11-23T02:20:42.0398174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj5wqoch/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0398446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0398572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0398851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0398985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0399171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0u0oo9e
2022-11-23T02:20:42.0399363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0u0oo9e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0399514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0399680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0399838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0399995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0400065Z ok (2.674s)
2022-11-23T02:20:42.0400070Z 
2022-11-23T02:20:42.0400271Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0400345Z Ran 1 test in 2.674s
2022-11-23T02:20:42.0400350Z 
2022-11-23T02:20:42.0400410Z OK
2022-11-23T02:20:42.0400415Z 
2022-11-23T02:20:42.0400485Z Generating XML reports...
2022-11-23T02:20:42.0400860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015001.xml
2022-11-23T02:20:42.0401139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0401261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0401543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0401679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0401868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr6n2i1i
2022-11-23T02:20:42.0402063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr6n2i1i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0402320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0402359Z 
2022-11-23T02:20:42.0402419Z Running tests...
2022-11-23T02:20:42.0402616Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0402861Z   test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13310
2022-11-23T02:20:42.0403014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13311
2022-11-23T02:20:42.0403167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13312
2022-11-23T02:20:42.0403318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13313
2022-11-23T02:20:42.0403621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0403743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0404024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0404148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0404334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2hb4oei
2022-11-23T02:20:42.0404527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2hb4oei/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0404801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0404991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0405282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0405416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0405599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpv_44gn
2022-11-23T02:20:42.0405783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpv_44gn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0406057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0406179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0406459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0406594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0406781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxhxonol
2022-11-23T02:20:42.0406978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxhxonol/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0407248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0407373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0407640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0407774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0407957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc4biwrp
2022-11-23T02:20:42.0408152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc4biwrp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0408315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0408481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0408642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0408840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0408895Z ok (2.578s)
2022-11-23T02:20:42.0408914Z 
2022-11-23T02:20:42.0409099Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0409174Z Ran 1 test in 2.578s
2022-11-23T02:20:42.0409179Z 
2022-11-23T02:20:42.0409239Z OK
2022-11-23T02:20:42.0409244Z 
2022-11-23T02:20:42.0409331Z Generating XML reports...
2022-11-23T02:20:42.0409699Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015006.xml
2022-11-23T02:20:42.0409975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0410126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0410413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0410536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0410723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9kivuj9
2022-11-23T02:20:42.0410920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9kivuj9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0411178Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0411184Z 
2022-11-23T02:20:42.0411256Z Running tests...
2022-11-23T02:20:42.0411453Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0411689Z   test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13537
2022-11-23T02:20:42.0411852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13538
2022-11-23T02:20:42.0412008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13539
2022-11-23T02:20:42.0412162Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13540
2022-11-23T02:20:42.0412424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0412545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0412827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0412964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0413147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnao7v0k1
2022-11-23T02:20:42.0413345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnao7v0k1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0413616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0413738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0414019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0414141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0414324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtotlaoh
2022-11-23T02:20:42.0414519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtotlaoh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0414790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0415015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0415297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0415432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0415671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg166asr
2022-11-23T02:20:42.0415854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg166asr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0416129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0416251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0416531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0416713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0416894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp409c4rcu
2022-11-23T02:20:42.0417086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp409c4rcu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0417249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0417411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0417559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0417720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0417790Z ok (2.480s)
2022-11-23T02:20:42.0417796Z 
2022-11-23T02:20:42.0417992Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0418069Z Ran 1 test in 2.481s
2022-11-23T02:20:42.0418074Z 
2022-11-23T02:20:42.0418133Z OK
2022-11-23T02:20:42.0418138Z 
2022-11-23T02:20:42.0418221Z Generating XML reports...
2022-11-23T02:20:42.0418590Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015010.xml
2022-11-23T02:20:42.0418854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0418977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0419255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0419389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0419573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwsw2bt4
2022-11-23T02:20:42.0419767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwsw2bt4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0420021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0420026Z 
2022-11-23T02:20:42.0420098Z Running tests...
2022-11-23T02:20:42.0420291Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0420529Z   test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13780
2022-11-23T02:20:42.0420685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13781
2022-11-23T02:20:42.0420838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13782
2022-11-23T02:20:42.0420990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13783
2022-11-23T02:20:42.0421263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0421385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0421664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0421797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0422000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75v9tud9
2022-11-23T02:20:42.0422193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75v9tud9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0422464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0422589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0422868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0423001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0423246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4vpqwj3
2022-11-23T02:20:42.0423437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4vpqwj3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0423710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0423819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0424097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0424228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0424410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ze34t9k
2022-11-23T02:20:42.0424600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ze34t9k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0424870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0424994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0425273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0425406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0425577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvonj2mrw
2022-11-23T02:20:42.0425771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvonj2mrw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0425935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0426098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0426261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0426419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0426485Z ok (2.575s)
2022-11-23T02:20:42.0426490Z 
2022-11-23T02:20:42.0426683Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0426748Z Ran 1 test in 2.575s
2022-11-23T02:20:42.0426753Z 
2022-11-23T02:20:42.0426812Z OK
2022-11-23T02:20:42.0426817Z 
2022-11-23T02:20:42.0426899Z Generating XML reports...
2022-11-23T02:20:42.0427270Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015015.xml
2022-11-23T02:20:42.0427541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0427662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0427943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0428078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0428250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzyqtu4g
2022-11-23T02:20:42.0428476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzyqtu4g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0428730Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0428735Z 
2022-11-23T02:20:42.0428807Z Running tests...
2022-11-23T02:20:42.0429000Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0429157Z   test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:42.0429504Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14023
2022-11-23T02:20:42.0429689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14024
2022-11-23T02:20:42.0429841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14025
2022-11-23T02:20:42.0429981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14026
2022-11-23T02:20:42.0430258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0430381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0430662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0430796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0430982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbn_a9md
2022-11-23T02:20:42.0431174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbn_a9md/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0431447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0431568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0431840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0431974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0432158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3hmuzq8
2022-11-23T02:20:42.0432351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3hmuzq8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0432622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0432742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0433022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0433155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0433325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2f26b_iz
2022-11-23T02:20:42.0433517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2f26b_iz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0433785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0433905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0434183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0434315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0434496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1marcal7
2022-11-23T02:20:42.0434690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1marcal7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0434852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0435032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0435194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0435356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0435532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0435707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0435882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0436085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0436385Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0436684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0436963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0437253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0437528Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0437795Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0438056Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0438313Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0438567Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0438832Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0439085Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0439340Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0439597Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0439851Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0440101Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0440345Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0440411Z ok (2.782s)
2022-11-23T02:20:42.0440418Z 
2022-11-23T02:20:42.0440650Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0440728Z Ran 1 test in 2.782s
2022-11-23T02:20:42.0440733Z 
2022-11-23T02:20:42.0440795Z OK
2022-11-23T02:20:42.0440799Z 
2022-11-23T02:20:42.0440882Z Generating XML reports...
2022-11-23T02:20:42.0441249Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015019.xml
2022-11-23T02:20:42.0441526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0441676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0441947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0442081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0442266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumvxfbrw
2022-11-23T02:20:42.0442465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumvxfbrw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0442721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0442727Z 
2022-11-23T02:20:42.0442798Z Running tests...
2022-11-23T02:20:42.0442993Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0443242Z   test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14262
2022-11-23T02:20:42.0443388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14263
2022-11-23T02:20:42.0443542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14264
2022-11-23T02:20:42.0443693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14265
2022-11-23T02:20:42.0443971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0444094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0444376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0444513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0444698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z1crj14
2022-11-23T02:20:42.0444959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z1crj14/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0445226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0445348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0445629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0445763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0445944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__19z0qk
2022-11-23T02:20:42.0446132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__19z0qk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0446404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0446528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0446807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0446927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0447144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27h79rkl
2022-11-23T02:20:42.0447337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27h79rkl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0447608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0447730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0448004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0448136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0448351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2fyrzlu
2022-11-23T02:20:42.0448530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2fyrzlu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0448693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0448858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0449017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0449178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0449353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0449530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0449705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0449878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0450167Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0450465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0450757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0451046Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0451116Z ok (2.777s)
2022-11-23T02:20:42.0451122Z 
2022-11-23T02:20:42.0451320Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0451396Z Ran 1 test in 2.777s
2022-11-23T02:20:42.0451401Z 
2022-11-23T02:20:42.0451460Z OK
2022-11-23T02:20:42.0451465Z 
2022-11-23T02:20:42.0451547Z Generating XML reports...
2022-11-23T02:20:42.0451906Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015024.xml
2022-11-23T02:20:42.0452180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0452301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0452582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0452715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0452899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtqu8l14
2022-11-23T02:20:42.0453094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtqu8l14/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0453349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0453354Z 
2022-11-23T02:20:42.0453426Z Running tests...
2022-11-23T02:20:42.0453609Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0453890Z   test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14497
2022-11-23T02:20:42.0454050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14498
2022-11-23T02:20:42.0454205Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14499
2022-11-23T02:20:42.0454357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14500
2022-11-23T02:20:42.0454632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0454781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0455167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0455289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0455475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zyxeh21
2022-11-23T02:20:42.0455668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zyxeh21/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0455940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0456061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0456343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0456478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0456663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi0maaqw
2022-11-23T02:20:42.0456856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi0maaqw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0457123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0457245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0457524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0457657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0457839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozq98ig9
2022-11-23T02:20:42.0458029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozq98ig9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0458299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0458422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0458690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0458822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0459007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjawsey_
2022-11-23T02:20:42.0459291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjawsey_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0459455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0459618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0459780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0459941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0460116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0460350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0460529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0460704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0461007Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0461303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0461647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0461940Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0462006Z ok (2.782s)
2022-11-23T02:20:42.0462012Z 
2022-11-23T02:20:42.0462210Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0462272Z Ran 1 test in 2.782s
2022-11-23T02:20:42.0462277Z 
2022-11-23T02:20:42.0462335Z OK
2022-11-23T02:20:42.0462340Z 
2022-11-23T02:20:42.0462423Z Generating XML reports...
2022-11-23T02:20:42.0462792Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015028.xml
2022-11-23T02:20:42.0463067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0463192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0463480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0463614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0463789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m74onf5
2022-11-23T02:20:42.0463985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m74onf5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0464240Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0464245Z 
2022-11-23T02:20:42.0464319Z Running tests...
2022-11-23T02:20:42.0464513Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0464766Z   test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14732
2022-11-23T02:20:42.0464925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14733
2022-11-23T02:20:42.0465080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14734
2022-11-23T02:20:42.0465233Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14735
2022-11-23T02:20:42.0465501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0465624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0465907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0466043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0466231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq1jsro1
2022-11-23T02:20:42.0466430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq1jsro1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0466707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0466830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0467148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0467270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0467454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ha24gnk
2022-11-23T02:20:42.0467650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ha24gnk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0467923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0468076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0468354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0468489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0468675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb04ebnlm
2022-11-23T02:20:42.0468857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb04ebnlm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0469128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0469250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0469528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0469660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0469845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxfzd9hc
2022-11-23T02:20:42.0470041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxfzd9hc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0470206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0470373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0470523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0470683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0470864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0471038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0471215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0471387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0471685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0471983Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0472278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0472559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0472625Z ok (2.776s)
2022-11-23T02:20:42.0472631Z 
2022-11-23T02:20:42.0472827Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0472905Z Ran 1 test in 2.776s
2022-11-23T02:20:42.0472910Z 
2022-11-23T02:20:42.0472969Z OK
2022-11-23T02:20:42.0472974Z 
2022-11-23T02:20:42.0473057Z Generating XML reports...
2022-11-23T02:20:42.0473426Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015033.xml
2022-11-23T02:20:42.0473740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0473852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0474135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0474273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0474460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1kz5w39
2022-11-23T02:20:42.0474660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1kz5w39/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0474942Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0474947Z 
2022-11-23T02:20:42.0475019Z Running tests...
2022-11-23T02:20:42.0475218Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0475473Z   test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14967
2022-11-23T02:20:42.0475615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14968
2022-11-23T02:20:42.0475769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14969
2022-11-23T02:20:42.0475920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14970
2022-11-23T02:20:42.0476194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0476319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0476599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0476733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0476921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71hup5bg
2022-11-23T02:20:42.0477191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0477300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0477579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0477714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0477896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpou7r086l
2022-11-23T02:20:42.0478094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71hup5bg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0478281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpou7r086l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0478557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0478680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0478950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0479082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0479265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhiqls_5
2022-11-23T02:20:42.0479460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhiqls_5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0479730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0479850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0480165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0480299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0480480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtjyy26t
2022-11-23T02:20:42.0480662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtjyy26t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0480825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0480985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0481176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0481335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0481510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0481685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0481859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0482157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0482443Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0482616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0482915Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0483213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0483280Z ok (2.680s)
2022-11-23T02:20:42.0483284Z 
2022-11-23T02:20:42.0483483Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0483556Z Ran 1 test in 2.681s
2022-11-23T02:20:42.0483561Z 
2022-11-23T02:20:42.0483620Z OK
2022-11-23T02:20:42.0483625Z 
2022-11-23T02:20:42.0483696Z Generating XML reports...
2022-11-23T02:20:42.0484066Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015038.xml
2022-11-23T02:20:42.0484341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0484467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0484748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0484881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0485151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82ge0a73
2022-11-23T02:20:42.0485346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82ge0a73/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0485605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0485610Z 
2022-11-23T02:20:42.0485669Z Running tests...
2022-11-23T02:20:42.0485864Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0486094Z   test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15202
2022-11-23T02:20:42.0486251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15203
2022-11-23T02:20:42.0486406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15204
2022-11-23T02:20:42.0486557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15205
2022-11-23T02:20:42.0486873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0486996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0487277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0487399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0487584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27lmpxv1
2022-11-23T02:20:42.0487780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27lmpxv1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0488081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0488204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0488489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0488622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0488803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8i9_g7wv
2022-11-23T02:20:42.0488982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8i9_g7wv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0489253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0489374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0489657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0489789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0489973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8h5d3e1
2022-11-23T02:20:42.0490168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8h5d3e1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0490437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0490558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0490823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0490956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0491140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8hmtztv
2022-11-23T02:20:42.0491332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8hmtztv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0491496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0491660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0491821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0491981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0492145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0492319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0492492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0492665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0492967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0493299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0493594Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0493886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0493951Z ok (2.777s)
2022-11-23T02:20:42.0493956Z 
2022-11-23T02:20:42.0494151Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0494244Z Ran 1 test in 2.777s
2022-11-23T02:20:42.0494249Z 
2022-11-23T02:20:42.0494309Z OK
2022-11-23T02:20:42.0494314Z 
2022-11-23T02:20:42.0494399Z Generating XML reports...
2022-11-23T02:20:42.0494767Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015042.xml
2022-11-23T02:20:42.0495161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0495284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0495566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0495700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0495874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjm5t8lyw
2022-11-23T02:20:42.0496070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjm5t8lyw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0496323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0496329Z 
2022-11-23T02:20:42.0496400Z Running tests...
2022-11-23T02:20:42.0496595Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0497309Z   test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.261s)
2022-11-23T02:20:42.0497316Z 
2022-11-23T02:20:42.0497510Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0497584Z Ran 1 test in 0.261s
2022-11-23T02:20:42.0497588Z 
2022-11-23T02:20:42.0497660Z OK (skipped=1)
2022-11-23T02:20:42.0497664Z 
2022-11-23T02:20:42.0497745Z Generating XML reports...
2022-11-23T02:20:42.0498101Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015047.xml
2022-11-23T02:20:42.0498375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0498500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0498781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0498914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0499100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbxfgvtk
2022-11-23T02:20:42.0499295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbxfgvtk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0499550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0499555Z 
2022-11-23T02:20:42.0499625Z Running tests...
2022-11-23T02:20:42.0499808Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0500096Z   test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15467
2022-11-23T02:20:42.0500255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15468
2022-11-23T02:20:42.0500408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15469
2022-11-23T02:20:42.0500560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15470
2022-11-23T02:20:42.0500837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0500958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0501279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0501401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0501586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1cik7dy
2022-11-23T02:20:42.0501781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1cik7dy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0502057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0502178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0502458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0502592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0502778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpysgo0qm9
2022-11-23T02:20:42.0502970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpysgo0qm9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0503232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0503357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0503638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0503771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0503955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mgva08u
2022-11-23T02:20:42.0504149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mgva08u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0504421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0504547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0504815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0504948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0505132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobt_qk5s
2022-11-23T02:20:42.0505322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobt_qk5s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0505484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0505648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0505810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0505973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0506038Z ok (2.580s)
2022-11-23T02:20:42.0506044Z 
2022-11-23T02:20:42.0506225Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0506299Z Ran 1 test in 2.580s
2022-11-23T02:20:42.0506304Z 
2022-11-23T02:20:42.0506364Z OK
2022-11-23T02:20:42.0506409Z 
2022-11-23T02:20:42.0506493Z Generating XML reports...
2022-11-23T02:20:42.0506861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015049.xml
2022-11-23T02:20:42.0507132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0507254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0507531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0507697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0507880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93zgik6z
2022-11-23T02:20:42.0508073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93zgik6z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0508330Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0508336Z 
2022-11-23T02:20:42.0508406Z Running tests...
2022-11-23T02:20:42.0508600Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0508856Z   test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15690
2022-11-23T02:20:42.0509011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15691
2022-11-23T02:20:42.0509164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15692
2022-11-23T02:20:42.0509304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15693
2022-11-23T02:20:42.0509575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0509696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0509978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0510109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0510291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp815ye29b
2022-11-23T02:20:42.0510485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp815ye29b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0510756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0510881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0511147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0511280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0511466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw8e31g9
2022-11-23T02:20:42.0511660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw8e31g9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0511931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0512051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0512329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0512463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0512634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsypflkuw
2022-11-23T02:20:42.0512826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsypflkuw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0513123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0513248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0513525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0513656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0513840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplywbgwno
2022-11-23T02:20:42.0514033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplywbgwno/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0514225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0514375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0514537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0514700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0515193Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0515331Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0515818Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0515956Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0516442Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0516575Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0517060Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0517195Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0517252Z ok (2.474s)
2022-11-23T02:20:42.0517257Z 
2022-11-23T02:20:42.0517451Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0517526Z Ran 1 test in 2.475s
2022-11-23T02:20:42.0517531Z 
2022-11-23T02:20:42.0517591Z OK
2022-11-23T02:20:42.0517596Z 
2022-11-23T02:20:42.0517679Z Generating XML reports...
2022-11-23T02:20:42.0518046Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015053.xml
2022-11-23T02:20:42.0518319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0518442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0518713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0518847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0519033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lq4suze
2022-11-23T02:20:42.0519229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lq4suze/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0519484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0519488Z 
2022-11-23T02:20:42.0519591Z Running tests...
2022-11-23T02:20:42.0519789Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0520031Z   test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15945
2022-11-23T02:20:42.0520187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15946
2022-11-23T02:20:42.0520328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15947
2022-11-23T02:20:42.0520481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15948
2022-11-23T02:20:42.0520785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0520910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0521194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0521329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0521511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_f9o4ga
2022-11-23T02:20:42.0521703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_f9o4ga/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0521965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0522087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0522370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0522505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0522690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1pvfx83
2022-11-23T02:20:42.0522884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1pvfx83/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0523153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0523275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0523555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0523676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0523858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6kat2yr
2022-11-23T02:20:42.0524050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6kat2yr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0524319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0524444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0524721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0524854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0525114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2g_6_3v
2022-11-23T02:20:42.0525294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2g_6_3v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0525456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0525622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0525783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0525945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0526197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0526376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0526549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0526852Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0527012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0527341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0527638Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0527930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0527998Z ok (2.675s)
2022-11-23T02:20:42.0528004Z 
2022-11-23T02:20:42.0528201Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0528275Z Ran 1 test in 2.675s
2022-11-23T02:20:42.0528280Z 
2022-11-23T02:20:42.0528339Z OK
2022-11-23T02:20:42.0528344Z 
2022-11-23T02:20:42.0528426Z Generating XML reports...
2022-11-23T02:20:42.0528781Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015058.xml
2022-11-23T02:20:42.0529056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0529178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0529456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0529591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0529786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3acv0d5
2022-11-23T02:20:42.0529982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3acv0d5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0530235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0530239Z 
2022-11-23T02:20:42.0530311Z Running tests...
2022-11-23T02:20:42.0530494Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0530747Z   test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16200
2022-11-23T02:20:42.0530902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16201
2022-11-23T02:20:42.0531059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16202
2022-11-23T02:20:42.0531212Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16203
2022-11-23T02:20:42.0531484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0531606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0531887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0532008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0532197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2acuoln
2022-11-23T02:20:42.0532392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2acuoln/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0532663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0532816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0533095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0533228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0533410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjoa387dj
2022-11-23T02:20:42.0533604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjoa387dj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0533889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0534009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0534288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0534423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0534605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7w80xf69
2022-11-23T02:20:42.0534794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7w80xf69/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0535198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0535319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0535586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0535721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0535903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuknkk4bq
2022-11-23T02:20:42.0536098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuknkk4bq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0536261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0536425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0536585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0536745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0536921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0537085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0537257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0537428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0537729Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0538026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0538318Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0538606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0538675Z ok (2.679s)
2022-11-23T02:20:42.0538680Z 
2022-11-23T02:20:42.0538874Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0538935Z Ran 1 test in 2.679s
2022-11-23T02:20:42.0538941Z 
2022-11-23T02:20:42.0539002Z OK
2022-11-23T02:20:42.0539006Z 
2022-11-23T02:20:42.0539089Z Generating XML reports...
2022-11-23T02:20:42.0539511Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015102.xml
2022-11-23T02:20:42.0539786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0539907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0540187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0540324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0540550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2bclp7t
2022-11-23T02:20:42.0540733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2bclp7t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0540991Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0540996Z 
2022-11-23T02:20:42.0541068Z Running tests...
2022-11-23T02:20:42.0541262Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0541502Z   test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16455
2022-11-23T02:20:42.0541658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16456
2022-11-23T02:20:42.0541812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16457
2022-11-23T02:20:42.0541963Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16458
2022-11-23T02:20:42.0542229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0542350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0542635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0542769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0542955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwe26p67
2022-11-23T02:20:42.0543151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwe26p67/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0543425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0543547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0543828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0543952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0544136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4ur506k
2022-11-23T02:20:42.0544331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4ur506k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0544605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0544727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0545009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0545142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0545325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlpso7d1
2022-11-23T02:20:42.0545510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlpso7d1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0545782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0545935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0546221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0546355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0546540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2foj5vw
2022-11-23T02:20:42.0546735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2foj5vw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0546898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0547091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0547240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0547398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0547577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0547752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0548057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0548228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0548398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0548693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0548991Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0549272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0549341Z ok (2.774s)
2022-11-23T02:20:42.0549346Z 
2022-11-23T02:20:42.0549541Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0549618Z Ran 1 test in 2.774s
2022-11-23T02:20:42.0549623Z 
2022-11-23T02:20:42.0549682Z OK
2022-11-23T02:20:42.0549687Z 
2022-11-23T02:20:42.0549768Z Generating XML reports...
2022-11-23T02:20:42.0550137Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015107.xml
2022-11-23T02:20:42.0550413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0550534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0550803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0550938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0551125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__ob7gdo
2022-11-23T02:20:42.0551318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__ob7gdo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0551572Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0551578Z 
2022-11-23T02:20:42.0551649Z Running tests...
2022-11-23T02:20:42.0551842Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0552091Z   test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16690
2022-11-23T02:20:42.0552235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16691
2022-11-23T02:20:42.0552415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16692
2022-11-23T02:20:42.0552569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16693
2022-11-23T02:20:42.0552844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0552966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0553247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0553380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0553609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptemg55at
2022-11-23T02:20:42.0553802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptemg55at/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0554061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0554185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0554464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0554599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0554782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6i49yqv
2022-11-23T02:20:42.0554973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6i49yqv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0555243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0555363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0555641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0555763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0555945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkimawja6
2022-11-23T02:20:42.0556141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkimawja6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0556414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0556536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0556814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0556948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0557128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ksrdq8s
2022-11-23T02:20:42.0557309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ksrdq8s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0557473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0557634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0557795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0557956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0558132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0558308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0558606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0558777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0559090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0559264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0559561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0559852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0559947Z ok (2.778s)
2022-11-23T02:20:42.0559952Z 
2022-11-23T02:20:42.0560147Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0560221Z Ran 1 test in 2.778s
2022-11-23T02:20:42.0560226Z 
2022-11-23T02:20:42.0560284Z OK
2022-11-23T02:20:42.0560289Z 
2022-11-23T02:20:42.0560371Z Generating XML reports...
2022-11-23T02:20:42.0560728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015112.xml
2022-11-23T02:20:42.0561002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0561123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0561403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0561536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0561724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk6wuf1f
2022-11-23T02:20:42.0561918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk6wuf1f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0562171Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0562176Z 
2022-11-23T02:20:42.0562248Z Running tests...
2022-11-23T02:20:42.0562430Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0562673Z   test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16925
2022-11-23T02:20:42.0562832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16926
2022-11-23T02:20:42.0562986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16927
2022-11-23T02:20:42.0563137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16928
2022-11-23T02:20:42.0563412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0563531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0563814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0563935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0564119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwknbtay
2022-11-23T02:20:42.0564311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwknbtay/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0564582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0564704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0565063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0565198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0565380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1jekviq
2022-11-23T02:20:42.0565612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1jekviq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0565870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0565993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0566273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0566404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0566586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht1n2s5y
2022-11-23T02:20:42.0566808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht1n2s5y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0567077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0567201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0567467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0567601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0567783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3ejkb3c
2022-11-23T02:20:42.0567976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3ejkb3c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0568138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0568303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0568465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0568627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0568805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0568969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0569143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0569316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0569614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0569914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0570206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0570497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0570563Z ok (2.779s)
2022-11-23T02:20:42.0570568Z 
2022-11-23T02:20:42.0570764Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0570826Z Ran 1 test in 2.779s
2022-11-23T02:20:42.0570832Z 
2022-11-23T02:20:42.0570890Z OK
2022-11-23T02:20:42.0570895Z 
2022-11-23T02:20:42.0570977Z Generating XML reports...
2022-11-23T02:20:42.0571347Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015116.xml
2022-11-23T02:20:42.0571620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0571741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0572020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0572183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0572354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a_ssunj
2022-11-23T02:20:42.0572546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a_ssunj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0572800Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0572805Z 
2022-11-23T02:20:42.0572907Z Running tests...
2022-11-23T02:20:42.0573101Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0573384Z   test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17160
2022-11-23T02:20:42.0573539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17161
2022-11-23T02:20:42.0573693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17162
2022-11-23T02:20:42.0573845Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17163
2022-11-23T02:20:42.0574106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0574229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0574511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0574648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0574834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmsok7fg
2022-11-23T02:20:42.0575148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmsok7fg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0575424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0575551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0575831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0575951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0576135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe20ok42h
2022-11-23T02:20:42.0576329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe20ok42h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0576601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0576724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0577006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0577143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0577327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7jxhijz
2022-11-23T02:20:42.0577508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7jxhijz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0577779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0577901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0578180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0578315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0578499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdt0tplua
2022-11-23T02:20:42.0578741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdt0tplua/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0578905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0579067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0579215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0579376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0579552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0579765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0579938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0580236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0580408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0580706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0581002Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0581282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0581350Z ok (2.779s)
2022-11-23T02:20:42.0581355Z 
2022-11-23T02:20:42.0581551Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0581626Z Ran 1 test in 2.779s
2022-11-23T02:20:42.0581631Z 
2022-11-23T02:20:42.0581691Z OK
2022-11-23T02:20:42.0581696Z 
2022-11-23T02:20:42.0581779Z Generating XML reports...
2022-11-23T02:20:42.0582152Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015121.xml
2022-11-23T02:20:42.0582424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0582536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0582819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0582954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0583142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfu1s6x6j
2022-11-23T02:20:42.0583336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfu1s6x6j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0583590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0583595Z 
2022-11-23T02:20:42.0583668Z Running tests...
2022-11-23T02:20:42.0583865Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0584117Z   test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17395
2022-11-23T02:20:42.0584261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17396
2022-11-23T02:20:42.0584416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17397
2022-11-23T02:20:42.0584568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17398
2022-11-23T02:20:42.0584845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0584967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0585281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0585416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0585599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevk6bhdp
2022-11-23T02:20:42.0585794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevk6bhdp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0586055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0586177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0586488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0586624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0586807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8brci4tz
2022-11-23T02:20:42.0587002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8brci4tz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0587271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0587393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0587659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0587794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0587974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06r_v7xp
2022-11-23T02:20:42.0588168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06r_v7xp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0588437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0588561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0588839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0588971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0589151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gk870cr
2022-11-23T02:20:42.0589330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gk870cr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0589495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0589661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0589822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0589981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0590159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0590334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0590507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0590665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0590966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0591267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0591563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0591883Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0591951Z ok (2.775s)
2022-11-23T02:20:42.0591956Z 
2022-11-23T02:20:42.0592152Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0592225Z Ran 1 test in 2.775s
2022-11-23T02:20:42.0592230Z 
2022-11-23T02:20:42.0592290Z OK
2022-11-23T02:20:42.0592294Z 
2022-11-23T02:20:42.0592364Z Generating XML reports...
2022-11-23T02:20:42.0592733Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015126.xml
2022-11-23T02:20:42.0593035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0593159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0593441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0593579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0593764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0zgcqfk
2022-11-23T02:20:42.0593960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0zgcqfk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0594214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0594219Z 
2022-11-23T02:20:42.0594278Z Running tests...
2022-11-23T02:20:42.0594472Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0594732Z   test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17630
2022-11-23T02:20:42.0594886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17631
2022-11-23T02:20:42.0595040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17632
2022-11-23T02:20:42.0595190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17633
2022-11-23T02:20:42.0595461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0595583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0595863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0595985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0596173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8irupu1_
2022-11-23T02:20:42.0596365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8irupu1_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0596638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0596759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0597037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0597169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0597352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fpz_3o9
2022-11-23T02:20:42.0597530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fpz_3o9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0597801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0597922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0598201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0598379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0598563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy4kj9n8
2022-11-23T02:20:42.0598753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy4kj9n8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0599023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0599145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0599410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0599585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0599766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a3l7dun
2022-11-23T02:20:42.0599959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a3l7dun/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0600121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0600284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0600443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0600605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0600769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0600947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0601121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0601422Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0601592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0601890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0602186Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0602476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0602545Z ok (2.677s)
2022-11-23T02:20:42.0602551Z 
2022-11-23T02:20:42.0602746Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0602808Z Ran 1 test in 2.678s
2022-11-23T02:20:42.0602813Z 
2022-11-23T02:20:42.0602874Z OK
2022-11-23T02:20:42.0602878Z 
2022-11-23T02:20:42.0602961Z Generating XML reports...
2022-11-23T02:20:42.0603329Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015130.xml
2022-11-23T02:20:42.0603603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0603724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0604003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0604136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0604308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wwj9ldj
2022-11-23T02:20:42.0604502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wwj9ldj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0604755Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0604760Z 
2022-11-23T02:20:42.0604859Z Running tests...
2022-11-23T02:20:42.0605118Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0605360Z   test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17865
2022-11-23T02:20:42.0605516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17866
2022-11-23T02:20:42.0605670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17867
2022-11-23T02:20:42.0605822Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17868
2022-11-23T02:20:42.0606124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0606247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0606531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0606668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0606851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv3xe270
2022-11-23T02:20:42.0607043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv3xe270/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0607311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0607433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0607703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0607836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0608018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflri7lnz
2022-11-23T02:20:42.0608213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflri7lnz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0608482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0608603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0608881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0609013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0609197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3eh3j5a
2022-11-23T02:20:42.0609377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3eh3j5a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0609646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0609767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0610047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0610182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0610363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes9du8zb
2022-11-23T02:20:42.0610552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes9du8zb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0610713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0610876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0611024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0611184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0611392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0611569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0611743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0612042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0612214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0612542Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0612840Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0613120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0613187Z ok (2.775s)
2022-11-23T02:20:42.0613192Z 
2022-11-23T02:20:42.0613389Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0613463Z Ran 1 test in 2.775s
2022-11-23T02:20:42.0613468Z 
2022-11-23T02:20:42.0613528Z OK
2022-11-23T02:20:42.0613533Z 
2022-11-23T02:20:42.0613615Z Generating XML reports...
2022-11-23T02:20:42.0613983Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015135.xml
2022-11-23T02:20:42.0614255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0614364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0614642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0614777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0615070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfq_s_8a
2022-11-23T02:20:42.0615264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfq_s_8a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0615518Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0615523Z 
2022-11-23T02:20:42.0615594Z Running tests...
2022-11-23T02:20:42.0615789Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0616039Z   test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18100
2022-11-23T02:20:42.0616182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18101
2022-11-23T02:20:42.0616339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18102
2022-11-23T02:20:42.0616489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18103
2022-11-23T02:20:42.0616763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0616887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0617169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0617304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0617491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztbls4qk
2022-11-23T02:20:42.0617675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztbls4qk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0617944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0618113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0618395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0618531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0618714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyyczui8
2022-11-23T02:20:42.0618909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyyczui8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0619219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0619339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0619606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0619740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0619923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjmk2k58
2022-11-23T02:20:42.0620116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjmk2k58/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0620387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0620509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0620788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0620925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0621109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtocijxf
2022-11-23T02:20:42.0621292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtocijxf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0621455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0621617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0621780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0621938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0622114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0622290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0622462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0622621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0622923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0623222Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0623515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0623808Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0623877Z ok (2.776s)
2022-11-23T02:20:42.0623882Z 
2022-11-23T02:20:42.0624076Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0624150Z Ran 1 test in 2.776s
2022-11-23T02:20:42.0624155Z 
2022-11-23T02:20:42.0624213Z OK
2022-11-23T02:20:42.0624218Z 
2022-11-23T02:20:42.0624288Z Generating XML reports...
2022-11-23T02:20:42.0624687Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015139.xml
2022-11-23T02:20:42.0624964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0625088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0625369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0625505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0625755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp0o5sz7
2022-11-23T02:20:42.0625950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp0o5sz7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0626204Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0626209Z 
2022-11-23T02:20:42.0626269Z Running tests...
2022-11-23T02:20:42.0626466Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0626705Z   test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18335
2022-11-23T02:20:42.0626859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18336
2022-11-23T02:20:42.0627011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18337
2022-11-23T02:20:42.0627164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18338
2022-11-23T02:20:42.0627439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0627561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0627832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0627967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0628150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc6o2j0g
2022-11-23T02:20:42.0628346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc6o2j0g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0628617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0628739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0629021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0629156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0629341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd99tqln
2022-11-23T02:20:42.0629525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd99tqln/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0629796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0629918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0630198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0630333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0630516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopm7dkhm
2022-11-23T02:20:42.0630714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopm7dkhm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0630983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0631131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0631396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0631530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0631715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwef8y1ek
2022-11-23T02:20:42.0631912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwef8y1ek/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0632075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0632267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0632428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0632587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0632642Z ok (2.576s)
2022-11-23T02:20:42.0632647Z 
2022-11-23T02:20:42.0632844Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0632918Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0632923Z 
2022-11-23T02:20:42.0632985Z OK
2022-11-23T02:20:42.0632989Z 
2022-11-23T02:20:42.0633072Z Generating XML reports...
2022-11-23T02:20:42.0633438Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015144.xml
2022-11-23T02:20:42.0633709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0633834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0634114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0634235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0634422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdtjcolk
2022-11-23T02:20:42.0634617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdtjcolk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0634871Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0634877Z 
2022-11-23T02:20:42.0634952Z Running tests...
2022-11-23T02:20:42.0635147Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0635385Z   test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18578
2022-11-23T02:20:42.0635542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18579
2022-11-23T02:20:42.0635685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18580
2022-11-23T02:20:42.0635839Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18581
2022-11-23T02:20:42.0636112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0636234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0636520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0636654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0636839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0afl3ft
2022-11-23T02:20:42.0637037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0afl3ft/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0637307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0637415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0637721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0637855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0638038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtbuxeji
2022-11-23T02:20:42.0638232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtbuxeji/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0649823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0650051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0650338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0650474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0650649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7it0zr8
2022-11-23T02:20:42.0650844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7it0zr8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0651113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0651236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0651515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0651651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0651834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizni1quc
2022-11-23T02:20:42.0652029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizni1quc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0652181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0652345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0652505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0652664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0652730Z ok (3.381s)
2022-11-23T02:20:42.0652737Z 
2022-11-23T02:20:42.0652931Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0653005Z Ran 1 test in 3.381s
2022-11-23T02:20:42.0653012Z 
2022-11-23T02:20:42.0653072Z OK
2022-11-23T02:20:42.0653076Z 
2022-11-23T02:20:42.0653158Z Generating XML reports...
2022-11-23T02:20:42.0653517Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015148.xml
2022-11-23T02:20:42.0653790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0653912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0654193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0654327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0654510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemq1c6ys
2022-11-23T02:20:42.0654703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemq1c6ys/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0655257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0655263Z 
2022-11-23T02:20:42.0655334Z Running tests...
2022-11-23T02:20:42.0655516Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0655835Z   test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18821
2022-11-23T02:20:42.0655992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18822
2022-11-23T02:20:42.0656143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18823
2022-11-23T02:20:42.0656294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18824
2022-11-23T02:20:42.0656571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0656747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0657027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0657148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0657335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d27e6ch
2022-11-23T02:20:42.0657528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d27e6ch/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0657803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0657924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0658205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0658337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0658521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu29dluo
2022-11-23T02:20:42.0658714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu29dluo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0658977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0659097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0659373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0659505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0659686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptu3ivyc
2022-11-23T02:20:42.0659877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptu3ivyc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0660148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0660268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0660535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0660669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0660851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvs90a5or
2022-11-23T02:20:42.0661041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvs90a5or/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0661202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0661364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0661524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0661687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0661862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0662052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0662228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0662399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0662698Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0662997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0663319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0663653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0664011Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #151 to worker2, as the agent is shutting down
2022-11-23T02:20:42.0664282Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0664567Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #150 to worker3, as the agent is shutting down
2022-11-23T02:20:42.0664824Z [W tensorpipe_agent.cpp:678] RPC agent for worker2 encountered error when sending response to request #151 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0665091Z [W tensorpipe_agent.cpp:939] RPC agent for worker2 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0665358Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0665618Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.0665901Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #152 to worker0, as the agent is shutting down
2022-11-23T02:20:42.0666158Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0666418Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0666635Z [E container.cpp:256] Could not release Dist Autograd Context on node 2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0666852Z [E container.cpp:256] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0667103Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0667355Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0667611Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0667903Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0668144Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0668356Z [E container.cpp:256] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.0668422Z ok (2.778s)
2022-11-23T02:20:42.0668427Z 
2022-11-23T02:20:42.0668624Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0668727Z Ran 1 test in 2.779s
2022-11-23T02:20:42.0668732Z 
2022-11-23T02:20:42.0668790Z OK
2022-11-23T02:20:42.0668795Z 
2022-11-23T02:20:42.0668876Z Generating XML reports...
2022-11-23T02:20:42.0669245Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015154.xml
2022-11-23T02:20:42.0669520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0669629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0669911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0670043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0670228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vsgkwlz
2022-11-23T02:20:42.0670423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vsgkwlz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0670678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0670683Z 
2022-11-23T02:20:42.0670754Z Running tests...
2022-11-23T02:20:42.0670952Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0671208Z   test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19076
2022-11-23T02:20:42.0671352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19077
2022-11-23T02:20:42.0671504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19078
2022-11-23T02:20:42.0671655Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19079
2022-11-23T02:20:42.0671930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0672052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0672329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0672462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0672647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxkf2u61
2022-11-23T02:20:42.0672830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxkf2u61/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0673103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0673221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0673503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0673637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0673823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph42oufy1
2022-11-23T02:20:42.0674015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph42oufy1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0674313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0674435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0674702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0674833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0675012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o8qf10k
2022-11-23T02:20:42.0675202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o8qf10k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0675499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0675619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0675899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0676035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0676204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjebn5zag
2022-11-23T02:20:42.0676397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjebn5zag/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0676560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0676722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0676883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0677044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0677110Z ok (2.476s)
2022-11-23T02:20:42.0677116Z 
2022-11-23T02:20:42.0677312Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0677387Z Ran 1 test in 2.476s
2022-11-23T02:20:42.0677392Z 
2022-11-23T02:20:42.0677439Z OK
2022-11-23T02:20:42.0677443Z 
2022-11-23T02:20:42.0677524Z Generating XML reports...
2022-11-23T02:20:42.0677893Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015158.xml
2022-11-23T02:20:42.0678165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0678286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0678565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0678700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0678884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70otslf_
2022-11-23T02:20:42.0679066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70otslf_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0679317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0679322Z 
2022-11-23T02:20:42.0679394Z Running tests...
2022-11-23T02:20:42.0679586Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0679821Z   test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19319
2022-11-23T02:20:42.0679975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19320
2022-11-23T02:20:42.0680130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19321
2022-11-23T02:20:42.0680280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19322
2022-11-23T02:20:42.0680554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0680695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0680981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0681113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0681356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghjk97du
2022-11-23T02:20:42.0681553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghjk97du/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0681825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0682000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0682319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0682442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0682676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn0fxno0
2022-11-23T02:20:42.0682909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn0fxno0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0683180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0683300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0683576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0683713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0683896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphck6q_kf
2022-11-23T02:20:42.0684089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphck6q_kf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0684348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0684469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0684747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0684880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0685139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfz2env4
2022-11-23T02:20:42.0685335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfz2env4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0685496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0685658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0685820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0685967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0686033Z ok (2.481s)
2022-11-23T02:20:42.0686038Z 
2022-11-23T02:20:42.0686235Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0686308Z Ran 1 test in 2.482s
2022-11-23T02:20:42.0686313Z 
2022-11-23T02:20:42.0686374Z OK
2022-11-23T02:20:42.0686378Z 
2022-11-23T02:20:42.0686459Z Generating XML reports...
2022-11-23T02:20:42.0686828Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015203.xml
2022-11-23T02:20:42.0687101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0687212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0687525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0687660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0687842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ulvuf45
2022-11-23T02:20:42.0688035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ulvuf45/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0688293Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0688298Z 
2022-11-23T02:20:42.0688397Z Running tests...
2022-11-23T02:20:42.0688593Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0688715Z   test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2022-11-23T02:20:42.0688913Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19542
2022-11-23T02:20:42.0689070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19543
2022-11-23T02:20:42.0689222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19544
2022-11-23T02:20:42.0689375Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19545
2022-11-23T02:20:42.0689646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0689769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0690052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0690190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0690362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45bngbhq
2022-11-23T02:20:42.0690556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45bngbhq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0690827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0690948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0713994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0714225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0714423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelpddejw
2022-11-23T02:20:42.0714621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelpddejw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0714956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0715080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0715372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0715507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0715694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbffn8326
2022-11-23T02:20:42.0715889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbffn8326/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0716163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0716290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0716562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0716697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0717024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4wakcj9
2022-11-23T02:20:42.0717222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4wakcj9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0717387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0717551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0717711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0717874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0717979Z ok (2.478s)
2022-11-23T02:20:42.0717985Z 
2022-11-23T02:20:42.0718185Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0718260Z Ran 1 test in 2.478s
2022-11-23T02:20:42.0718265Z 
2022-11-23T02:20:42.0718323Z OK
2022-11-23T02:20:42.0718327Z 
2022-11-23T02:20:42.0718414Z Generating XML reports...
2022-11-23T02:20:42.0718791Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015207.xml
2022-11-23T02:20:42.0719065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0719189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0719468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0719590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0719774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_fiup85
2022-11-23T02:20:42.0719966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_fiup85/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0720220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0720227Z 
2022-11-23T02:20:42.0720301Z Running tests...
2022-11-23T02:20:42.0720497Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0720742Z   test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19785
2022-11-23T02:20:42.0720900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19786
2022-11-23T02:20:42.0721040Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19787
2022-11-23T02:20:42.0721195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19788
2022-11-23T02:20:42.0721471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0721593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0721876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0722009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0722190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lba3ivx
2022-11-23T02:20:42.0722382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lba3ivx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0722654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0722763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0723042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0723176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0723362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwsogy_9
2022-11-23T02:20:42.0723595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwsogy_9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0723865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0723985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0724264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0724396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0724594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j81on5r
2022-11-23T02:20:42.0724787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j81on5r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0725159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0725286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0725568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0725700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0725881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_tquxh8
2022-11-23T02:20:42.0726073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_tquxh8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0726226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0726393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0726555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0726718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0727211Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0727347Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0727834Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0727972Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0728456Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0728592Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0729087Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2328: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-11-23T02:20:42.0729208Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-11-23T02:20:42.0729275Z ok (2.578s)
2022-11-23T02:20:42.0729281Z 
2022-11-23T02:20:42.0729478Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0729558Z Ran 1 test in 2.578s
2022-11-23T02:20:42.0729563Z 
2022-11-23T02:20:42.0729627Z OK
2022-11-23T02:20:42.0729632Z 
2022-11-23T02:20:42.0729713Z Generating XML reports...
2022-11-23T02:20:42.0730112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015211.xml
2022-11-23T02:20:42.0730382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0730492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0730770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0730902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0731086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1069u2z
2022-11-23T02:20:42.0731310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1069u2z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0731560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0731565Z 
2022-11-23T02:20:42.0731635Z Running tests...
2022-11-23T02:20:42.0731832Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0732090Z   test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20040
2022-11-23T02:20:42.0732234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20041
2022-11-23T02:20:42.0732390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20042
2022-11-23T02:20:42.0732541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20043
2022-11-23T02:20:42.0732815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0732939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0733219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0733354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0733538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt5vz85u
2022-11-23T02:20:42.0733722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt5vz85u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0733991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0734111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0734394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0734529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0734714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6_03bz1
2022-11-23T02:20:42.0735022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6_03bz1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0735298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0735419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0735685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0735823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0736005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_swwt32
2022-11-23T02:20:42.0736200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_swwt32/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0736470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0736592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0736957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0737095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0737282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw52sgu92
2022-11-23T02:20:42.0737464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw52sgu92/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0737631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0737794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0737988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0738150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0738328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0738505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0738678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0738838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0739141Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0739438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0739731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0740019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0740089Z ok (2.675s)
2022-11-23T02:20:42.0740095Z 
2022-11-23T02:20:42.0740292Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0740367Z Ran 1 test in 2.675s
2022-11-23T02:20:42.0740372Z 
2022-11-23T02:20:42.0740432Z OK
2022-11-23T02:20:42.0740436Z 
2022-11-23T02:20:42.0740506Z Generating XML reports...
2022-11-23T02:20:42.0740875Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015216.xml
2022-11-23T02:20:42.0741146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0741270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0741550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0741684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0741872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6vfd5ew
2022-11-23T02:20:42.0742069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6vfd5ew/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0742324Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0742329Z 
2022-11-23T02:20:42.0742388Z Running tests...
2022-11-23T02:20:42.0742581Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0742848Z   test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20275
2022-11-23T02:20:42.0743005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20276
2022-11-23T02:20:42.0743157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20277
2022-11-23T02:20:42.0743337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20278
2022-11-23T02:20:42.0743610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0743732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0744016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0744137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0744320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvg2mutim
2022-11-23T02:20:42.0744557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvg2mutim/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0744832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0744955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0745235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0745368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0745548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgsoj88c
2022-11-23T02:20:42.0745731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgsoj88c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0745999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0746123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0746403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0746534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0746718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7y035v0
2022-11-23T02:20:42.0746909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7y035v0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0747180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0747304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0747571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0747706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0747889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy75ah8fp
2022-11-23T02:20:42.0748081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy75ah8fp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0748245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0748408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0748571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0748737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0748901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.0749075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.0749375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0749546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.0749744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.0750036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0750330Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0750617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.0750684Z ok (2.785s)
2022-11-23T02:20:42.0750689Z 
2022-11-23T02:20:42.0750913Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0750974Z Ran 1 test in 2.785s
2022-11-23T02:20:42.0750980Z 
2022-11-23T02:20:42.0751037Z OK
2022-11-23T02:20:42.0751042Z 
2022-11-23T02:20:42.0751124Z Generating XML reports...
2022-11-23T02:20:42.0751495Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015220.xml
2022-11-23T02:20:42.0751769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0751890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0752171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0752304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0752475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8a9qmxk
2022-11-23T02:20:42.0752670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8a9qmxk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0752921Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0752926Z 
2022-11-23T02:20:42.0752995Z Running tests...
2022-11-23T02:20:42.0753189Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0753418Z   test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20510
2022-11-23T02:20:42.0753574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20511
2022-11-23T02:20:42.0753727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20512
2022-11-23T02:20:42.0753880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20513
2022-11-23T02:20:42.0754140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0754265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0754546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0754681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0754863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplns5asuq
2022-11-23T02:20:42.0755058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplns5asuq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0755325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0755447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0755713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0755847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0756029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4x8cj9_
2022-11-23T02:20:42.0756221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4x8cj9_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0756513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0756634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0756910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0757043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0757226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwz65qzk
2022-11-23T02:20:42.0757434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwz65qzk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0757701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0757822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0758100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0758232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0758414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnieznyjz
2022-11-23T02:20:42.0758607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnieznyjz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0758768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0758918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0759083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0759241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0759307Z ok (2.479s)
2022-11-23T02:20:42.0759312Z 
2022-11-23T02:20:42.0759509Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0759584Z Ran 1 test in 2.480s
2022-11-23T02:20:42.0759588Z 
2022-11-23T02:20:42.0759646Z OK
2022-11-23T02:20:42.0759651Z 
2022-11-23T02:20:42.0759734Z Generating XML reports...
2022-11-23T02:20:42.0760103Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015225.xml
2022-11-23T02:20:42.0760363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0760488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0760766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0760899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0761083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeoyumvyk
2022-11-23T02:20:42.0761275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeoyumvyk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0761526Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0761532Z 
2022-11-23T02:20:42.0761601Z Running tests...
2022-11-23T02:20:42.0761781Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0762022Z   test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20753
2022-11-23T02:20:42.0762179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20754
2022-11-23T02:20:42.0762332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20755
2022-11-23T02:20:42.0762483Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20756
2022-11-23T02:20:42.0762781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0762904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0763182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0763315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0763487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtwocbd7
2022-11-23T02:20:42.0763679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtwocbd7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0763980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0764101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0764458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0764629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0764813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_j8gel2
2022-11-23T02:20:42.0765077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_j8gel2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0765358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0765466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0765749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0765882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0766064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ibq41is
2022-11-23T02:20:42.0766256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ibq41is/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0766528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0766648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0766925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0767043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0767227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58r2_u3y
2022-11-23T02:20:42.0767415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58r2_u3y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0767575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0767741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0767900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0768060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0768125Z ok (2.575s)
2022-11-23T02:20:42.0768131Z 
2022-11-23T02:20:42.0768329Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0768392Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0768397Z 
2022-11-23T02:20:42.0768455Z OK
2022-11-23T02:20:42.0768462Z 
2022-11-23T02:20:42.0768544Z Generating XML reports...
2022-11-23T02:20:42.0768912Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015230.xml
2022-11-23T02:20:42.0769180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0769343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0769621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0769756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0769927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg013vzf4
2022-11-23T02:20:42.0770118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg013vzf4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0770371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0770404Z 
2022-11-23T02:20:42.0770477Z Running tests...
2022-11-23T02:20:42.0770672Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0770909Z   test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20976
2022-11-23T02:20:42.0771066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20977
2022-11-23T02:20:42.0771220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20978
2022-11-23T02:20:42.0771372Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20979
2022-11-23T02:20:42.0771635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0771760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0772040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0772177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0772361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8q9ia88
2022-11-23T02:20:42.0772556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8q9ia88/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0772827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0772948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0773213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0773345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0773525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_c_tv4f
2022-11-23T02:20:42.0773718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_c_tv4f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0773986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0774105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0774380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0774514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0774694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37vtiyax
2022-11-23T02:20:42.0774989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37vtiyax/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0775263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0775387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0775664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0775798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0776036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahgd38y7
2022-11-23T02:20:42.0776231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahgd38y7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0776394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0776545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0776710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0776870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0776974Z ok (2.678s)
2022-11-23T02:20:42.0776979Z 
2022-11-23T02:20:42.0777176Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0777253Z Ran 1 test in 2.679s
2022-11-23T02:20:42.0777257Z 
2022-11-23T02:20:42.0777320Z OK
2022-11-23T02:20:42.0777324Z 
2022-11-23T02:20:42.0777408Z Generating XML reports...
2022-11-23T02:20:42.0777778Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015234.xml
2022-11-23T02:20:42.0778038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0778160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0778439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0778571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0778753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqo4n2dv
2022-11-23T02:20:42.0778948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqo4n2dv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0779200Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0779206Z 
2022-11-23T02:20:42.0779278Z Running tests...
2022-11-23T02:20:42.0779460Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0779704Z   test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21199
2022-11-23T02:20:42.0779859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21200
2022-11-23T02:20:42.0780013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21201
2022-11-23T02:20:42.0780166Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21202
2022-11-23T02:20:42.0780442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0780563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0780845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0780980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0781155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r8l8apb
2022-11-23T02:20:42.0781430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0781553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0781836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0781970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0782156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkamr77n
2022-11-23T02:20:42.0782457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0782578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0782857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0782976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0783157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_w4cfqm
2022-11-23T02:20:42.0783349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r8l8apb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0783570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkamr77n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0783755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_w4cfqm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0784028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0784150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0784431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0784552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0784735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvaxwgxn
2022-11-23T02:20:42.0784929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvaxwgxn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0785091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0785250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0785408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0785569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0785633Z ok (3.578s)
2022-11-23T02:20:42.0785638Z 
2022-11-23T02:20:42.0785829Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0785891Z Ran 1 test in 3.578s
2022-11-23T02:20:42.0785896Z 
2022-11-23T02:20:42.0785954Z OK
2022-11-23T02:20:42.0785959Z 
2022-11-23T02:20:42.0786039Z Generating XML reports...
2022-11-23T02:20:42.0786405Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015238.xml
2022-11-23T02:20:42.0786680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0786802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0787081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0787215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0787389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pyr11_f
2022-11-23T02:20:42.0787584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pyr11_f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0787837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0787842Z 
2022-11-23T02:20:42.0787911Z Running tests...
2022-11-23T02:20:42.0788103Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0788335Z   test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21442
2022-11-23T02:20:42.0788489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21443
2022-11-23T02:20:42.0788640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21444
2022-11-23T02:20:42.0788817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21445
2022-11-23T02:20:42.0789078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0789198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0789476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0789610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0789833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g38ncfc
2022-11-23T02:20:42.0790028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g38ncfc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0790297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0790420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0790687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0790818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0791001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxlgt6rq
2022-11-23T02:20:42.0791195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxlgt6rq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0791466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0791589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0791866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0792002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0792187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodvka69r
2022-11-23T02:20:42.0792367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodvka69r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0792636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0792755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0793037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0793171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0793352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjs_om5v
2022-11-23T02:20:42.0793542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjs_om5v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0793704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0793853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0794011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0794171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0794236Z ok (2.679s)
2022-11-23T02:20:42.0794241Z 
2022-11-23T02:20:42.0794438Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0794516Z Ran 1 test in 2.679s
2022-11-23T02:20:42.0794521Z 
2022-11-23T02:20:42.0794582Z OK
2022-11-23T02:20:42.0794586Z 
2022-11-23T02:20:42.0794670Z Generating XML reports...
2022-11-23T02:20:42.0795039Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015244.xml
2022-11-23T02:20:42.0795326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0795448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0795726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0795860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0796044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9tlqqjh
2022-11-23T02:20:42.0796264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9tlqqjh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0796517Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0796522Z 
2022-11-23T02:20:42.0796591Z Running tests...
2022-11-23T02:20:42.0796786Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0797027Z   test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21685
2022-11-23T02:20:42.0797183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21686
2022-11-23T02:20:42.0797337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21687
2022-11-23T02:20:42.0797488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21688
2022-11-23T02:20:42.0797760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0797881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0798160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0798292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0798465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrxgfxnj
2022-11-23T02:20:42.0798660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrxgfxnj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0798928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0799048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0799326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0799461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0799647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyaw3tp4
2022-11-23T02:20:42.0799838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyaw3tp4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0800109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0800218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0800497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0800630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0800813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqnauso9
2022-11-23T02:20:42.0801005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqnauso9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0801277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0801398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0801706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0801828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0802009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75b_lze5
2022-11-23T02:20:42.0802201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75b_lze5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0802365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0802526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0802714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0802871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0802936Z ok (2.779s)
2022-11-23T02:20:42.0802941Z 
2022-11-23T02:20:42.0803137Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0803199Z Ran 1 test in 2.779s
2022-11-23T02:20:42.0803204Z 
2022-11-23T02:20:42.0803263Z OK
2022-11-23T02:20:42.0803268Z 
2022-11-23T02:20:42.0803351Z Generating XML reports...
2022-11-23T02:20:42.0803718Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015249.xml
2022-11-23T02:20:42.0803992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0804115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0804394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0804526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0804698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu9jdal2
2022-11-23T02:20:42.0804964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu9jdal2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0805228Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0805233Z 
2022-11-23T02:20:42.0805332Z Running tests...
2022-11-23T02:20:42.0805596Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0805837Z   test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21928
2022-11-23T02:20:42.0805998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21929
2022-11-23T02:20:42.0806152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21930
2022-11-23T02:20:42.0806307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21931
2022-11-23T02:20:42.0806570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0806692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0806974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0807108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0807291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_sygppa
2022-11-23T02:20:42.0807487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_sygppa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0807762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0807884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0808162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0808321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0808507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6cw9_uu
2022-11-23T02:20:42.0808700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6cw9_uu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0808974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0809097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0809404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0809535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0809720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupbxtjsx
2022-11-23T02:20:42.0809906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupbxtjsx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0810175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0810295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0810572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0810705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0810888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgnqnprk
2022-11-23T02:20:42.0811081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgnqnprk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0811245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0811409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0811558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0811720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0811785Z ok (2.574s)
2022-11-23T02:20:42.0811790Z 
2022-11-23T02:20:42.0811984Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0812059Z Ran 1 test in 2.574s
2022-11-23T02:20:42.0812064Z 
2022-11-23T02:20:42.0812123Z OK
2022-11-23T02:20:42.0812128Z 
2022-11-23T02:20:42.0812208Z Generating XML reports...
2022-11-23T02:20:42.0812579Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015253.xml
2022-11-23T02:20:42.0812842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0812964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0813244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0813379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0813562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzx9n5qx
2022-11-23T02:20:42.0813754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzx9n5qx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0814009Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0814017Z 
2022-11-23T02:20:42.0814086Z Running tests...
2022-11-23T02:20:42.0814284Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0815149Z   test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T02:20:42.0815173Z 
2022-11-23T02:20:42.0815356Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0815432Z Ran 1 test in 0.255s
2022-11-23T02:20:42.0815436Z 
2022-11-23T02:20:42.0815506Z OK (skipped=1)
2022-11-23T02:20:42.0815511Z 
2022-11-23T02:20:42.0815594Z Generating XML reports...
2022-11-23T02:20:42.0815964Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015258.xml
2022-11-23T02:20:42.0816281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0816406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0816688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0816823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0816996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66dtjd7q
2022-11-23T02:20:42.0817191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66dtjd7q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0817443Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0817449Z 
2022-11-23T02:20:42.0817520Z Running tests...
2022-11-23T02:20:42.0817716Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0817960Z   test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22161
2022-11-23T02:20:42.0818114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22162
2022-11-23T02:20:42.0818268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22163
2022-11-23T02:20:42.0818408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22164
2022-11-23T02:20:42.0818685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0818807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0819088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0819223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0819408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2xo2xaf
2022-11-23T02:20:42.0819602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2xo2xaf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0819877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0820000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0820268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0820400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0820580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp891i5iym
2022-11-23T02:20:42.0820772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp891i5iym/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0821045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0821166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0821469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0821604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0821775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a4f5wii
2022-11-23T02:20:42.0821966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a4f5wii/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0822236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0822357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0822665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0822799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0822981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mjjc1w2
2022-11-23T02:20:42.0823175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mjjc1w2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0823338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0823488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0823650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0823810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0824602Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0824701Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.0824837Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0824928Z Traceback (most recent call last):
2022-11-23T02:20:42.0825191Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0825323Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0825591Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0825685Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0825941Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0826018Z     self.optim.step()
2022-11-23T02:20:42.0826251Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0826331Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0826623Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0826735Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0826818Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0826823Z 
2022-11-23T02:20:42.0827606Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0827702Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.0827835Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0827927Z Traceback (most recent call last):
2022-11-23T02:20:42.0828219Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0828353Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0828634Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0828726Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0828964Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0829042Z     self.optim.step()
2022-11-23T02:20:42.0829279Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0829396Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0829687Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0829797Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0829896Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0829902Z 
2022-11-23T02:20:42.0830682Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0831455Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0831547Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.0831682Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0831759Z Traceback (most recent call last):
2022-11-23T02:20:42.0832018Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0832150Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0832428Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0832522Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0832775Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0832855Z     self.optim.step()
2022-11-23T02:20:42.0833090Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0833159Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0833452Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0833562Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0833659Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0833664Z 
2022-11-23T02:20:42.0833755Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.0833887Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0833977Z Traceback (most recent call last):
2022-11-23T02:20:42.0834224Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0834362Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0834636Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0834730Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0834980Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0835088Z     self.optim.step()
2022-11-23T02:20:42.0835322Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0835405Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0835685Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0835794Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0835890Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0835896Z 
2022-11-23T02:20:42.0835984Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.0836194Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0836285Z Traceback (most recent call last):
2022-11-23T02:20:42.0836547Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0836680Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0836949Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0837042Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0837293Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0837374Z     self.optim.step()
2022-11-23T02:20:42.0837609Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0837691Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0837981Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0838098Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0838181Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0838187Z 
2022-11-23T02:20:42.0838275Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.0838411Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0838503Z Traceback (most recent call last):
2022-11-23T02:20:42.0838759Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0838893Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0839170Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0839265Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0839501Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0839584Z     self.optim.step()
2022-11-23T02:20:42.0839817Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0839897Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0840189Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0840302Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0840395Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0840400Z 
2022-11-23T02:20:42.0840488Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.0840611Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0840701Z Traceback (most recent call last):
2022-11-23T02:20:42.0840959Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0841089Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0841367Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0841459Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0841709Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0841789Z     self.optim.step()
2022-11-23T02:20:42.0842039Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0842123Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0842409Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0842519Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0842612Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0842617Z 
2022-11-23T02:20:42.0842703Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.0842867Z ValueError('Error running optimizer.')
2022-11-23T02:20:42.0842944Z Traceback (most recent call last):
2022-11-23T02:20:42.0843199Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0843329Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0843608Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 99, in _local_optimizer_step
2022-11-23T02:20:42.0843700Z     local_optim.step(autograd_ctx_id)
2022-11-23T02:20:42.0843947Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 89, in step
2022-11-23T02:20:42.0844026Z     self.optim.step()
2022-11-23T02:20:42.0844261Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 140, in wrapper
2022-11-23T02:20:42.0844329Z     out = func(*args, **kwargs)
2022-11-23T02:20:42.0844617Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-11-23T02:20:42.0844729Z     raise ValueError("Error running optimizer.")
2022-11-23T02:20:42.0844821Z ValueError: Error running optimizer.
2022-11-23T02:20:42.0844826Z 
2022-11-23T02:20:42.0844960Z ok (2.680s)
2022-11-23T02:20:42.0844968Z 
2022-11-23T02:20:42.0845174Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0845251Z Ran 1 test in 2.680s
2022-11-23T02:20:42.0845256Z 
2022-11-23T02:20:42.0845317Z OK
2022-11-23T02:20:42.0845322Z 
2022-11-23T02:20:42.0845404Z Generating XML reports...
2022-11-23T02:20:42.0845768Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015300.xml
2022-11-23T02:20:42.0846046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0846170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0846456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0846593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0846779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps35qpyx6
2022-11-23T02:20:42.0846979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps35qpyx6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0847235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0847241Z 
2022-11-23T02:20:42.0847299Z Running tests...
2022-11-23T02:20:42.0847496Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0847756Z   test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22404
2022-11-23T02:20:42.0847913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22405
2022-11-23T02:20:42.0848067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22406
2022-11-23T02:20:42.0848219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22407
2022-11-23T02:20:42.0848528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0848651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0848930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0849051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0849234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb45w9_0q
2022-11-23T02:20:42.0849427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb45w9_0q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0849739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0849858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0850140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0850273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0850454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt_aekwq
2022-11-23T02:20:42.0850634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt_aekwq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0850906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0851027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0851303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0851440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0851622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4no22r_
2022-11-23T02:20:42.0851817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4no22r_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0852086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0852209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0852476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0852609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0852791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6w46fan8
2022-11-23T02:20:42.0852985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6w46fan8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0853150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0853312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0853473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0853632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0854457Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0854554Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.0854682Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0854772Z Traceback (most recent call last):
2022-11-23T02:20:42.0855200Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0855336Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0855613Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0855748Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0856008Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0856073Z     **kwargs)
2022-11-23T02:20:42.0856356Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0856512Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0856609Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0856614Z 
2022-11-23T02:20:42.0857431Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0857524Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.0857661Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0857751Z Traceback (most recent call last):
2022-11-23T02:20:42.0858015Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0858149Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0858410Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0858542Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0858800Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0858865Z     **kwargs)
2022-11-23T02:20:42.0859157Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0859266Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0859364Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0859369Z 
2022-11-23T02:20:42.0860184Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0861005Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-11-23T02:20:42.0861094Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.0861230Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0861307Z Traceback (most recent call last):
2022-11-23T02:20:42.0861566Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0861702Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0861975Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0862132Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0862387Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0862450Z     **kwargs)
2022-11-23T02:20:42.0862743Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0862845Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0862941Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0862946Z 
2022-11-23T02:20:42.0863034Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.0863198Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0863288Z Traceback (most recent call last):
2022-11-23T02:20:42.0863550Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0863681Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0863958Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0864079Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0864333Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0864396Z     **kwargs)
2022-11-23T02:20:42.0864690Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0864801Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0864898Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0864902Z 
2022-11-23T02:20:42.0864989Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.0865124Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0865202Z Traceback (most recent call last):
2022-11-23T02:20:42.0865462Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0865594Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0865864Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0865994Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0866248Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0866312Z     **kwargs)
2022-11-23T02:20:42.0866604Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0866706Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0866801Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0866806Z 
2022-11-23T02:20:42.0866895Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.0867030Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0867120Z Traceback (most recent call last):
2022-11-23T02:20:42.0867378Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0867510Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0867779Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0867897Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0868151Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0868218Z     **kwargs)
2022-11-23T02:20:42.0868511Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0868623Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0868720Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0868756Z 
2022-11-23T02:20:42.0868844Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.0868982Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0869060Z Traceback (most recent call last):
2022-11-23T02:20:42.0869315Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0869448Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0869719Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0869875Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0870125Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0870187Z     **kwargs)
2022-11-23T02:20:42.0870485Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0870585Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0870684Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0870689Z 
2022-11-23T02:20:42.0870778Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.0870917Z ValueError('Error creating optimizer.')
2022-11-23T02:20:42.0871007Z Traceback (most recent call last):
2022-11-23T02:20:42.0871265Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.0871394Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.0871665Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 94, in _new_local_optimizer
2022-11-23T02:20:42.0871784Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-11-23T02:20:42.0872038Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 81, in __init__
2022-11-23T02:20:42.0872101Z     **kwargs)
2022-11-23T02:20:42.0872394Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-11-23T02:20:42.0872506Z     raise ValueError("Error creating optimizer.")
2022-11-23T02:20:42.0872603Z ValueError: Error creating optimizer.
2022-11-23T02:20:42.0872608Z 
2022-11-23T02:20:42.0872673Z ok (2.476s)
2022-11-23T02:20:42.0872678Z 
2022-11-23T02:20:42.0872871Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0872933Z Ran 1 test in 2.476s
2022-11-23T02:20:42.0872940Z 
2022-11-23T02:20:42.0873000Z OK
2022-11-23T02:20:42.0873005Z 
2022-11-23T02:20:42.0873088Z Generating XML reports...
2022-11-23T02:20:42.0873462Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015304.xml
2022-11-23T02:20:42.0873736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0873859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0874139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0874273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0874447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueqjhtdx
2022-11-23T02:20:42.0874641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueqjhtdx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0874895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0874900Z 
2022-11-23T02:20:42.0874971Z Running tests...
2022-11-23T02:20:42.0875165Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0875894Z   test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.254s)
2022-11-23T02:20:42.0875901Z 
2022-11-23T02:20:42.0876096Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0876171Z Ran 1 test in 0.254s
2022-11-23T02:20:42.0876176Z 
2022-11-23T02:20:42.0876244Z OK (skipped=1)
2022-11-23T02:20:42.0876249Z 
2022-11-23T02:20:42.0876361Z Generating XML reports...
2022-11-23T02:20:42.0876722Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015309.xml
2022-11-23T02:20:42.0876998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0877122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0877406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0877538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0877722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfp7j5me
2022-11-23T02:20:42.0877919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfp7j5me/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0878174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0878181Z 
2022-11-23T02:20:42.0878253Z Running tests...
2022-11-23T02:20:42.0878434Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0878674Z   test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22637
2022-11-23T02:20:42.0878833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22638
2022-11-23T02:20:42.0878986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22639
2022-11-23T02:20:42.0879139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22640
2022-11-23T02:20:42.0879412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0879536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0879819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0879953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0880124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbelx3w01
2022-11-23T02:20:42.0880317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbelx3w01/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0880591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0880712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0880991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0881125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0881308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpm4qnhe
2022-11-23T02:20:42.0881507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpm4qnhe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0881762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0881886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0882196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0882334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0882516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22m_plc1
2022-11-23T02:20:42.0882705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22m_plc1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0882975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0883141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0883423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0883543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0883724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30f0vy11
2022-11-23T02:20:42.0883914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30f0vy11/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0884076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0884236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0884396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0884555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0884622Z ok (2.576s)
2022-11-23T02:20:42.0884627Z 
2022-11-23T02:20:42.0884809Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0884881Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0884886Z 
2022-11-23T02:20:42.0885010Z OK
2022-11-23T02:20:42.0885016Z 
2022-11-23T02:20:42.0885102Z Generating XML reports...
2022-11-23T02:20:42.0885485Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015311.xml
2022-11-23T02:20:42.0885758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0885882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0886160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0886295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0886465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1usn5i_j
2022-11-23T02:20:42.0886661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1usn5i_j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0886917Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0886922Z 
2022-11-23T02:20:42.0886992Z Running tests...
2022-11-23T02:20:42.0887188Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0887481Z   test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22870
2022-11-23T02:20:42.0887637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22871
2022-11-23T02:20:42.0887793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22872
2022-11-23T02:20:42.0887937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22873
2022-11-23T02:20:42.0888215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0888339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0888658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0888790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0888979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszsmbfhp
2022-11-23T02:20:42.0889175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszsmbfhp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0889445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0889596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0889863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0889996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0890181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx42z2_lo
2022-11-23T02:20:42.0890370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx42z2_lo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0890640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0890761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0891042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0891177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0891360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y_2uztk
2022-11-23T02:20:42.0891539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y_2uztk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0891810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0891932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0892211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0892342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0892524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoj3oy_j
2022-11-23T02:20:42.0892717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoj3oy_j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0892881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0893032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0893190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0893354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0893419Z ok (2.581s)
2022-11-23T02:20:42.0893424Z 
2022-11-23T02:20:42.0893616Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0893688Z Ran 1 test in 2.581s
2022-11-23T02:20:42.0893693Z 
2022-11-23T02:20:42.0893750Z OK
2022-11-23T02:20:42.0893755Z 
2022-11-23T02:20:42.0893835Z Generating XML reports...
2022-11-23T02:20:42.0894203Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015315.xml
2022-11-23T02:20:42.0894479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0894600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0894990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0895174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0895360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbrb3rmy
2022-11-23T02:20:42.0895556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbrb3rmy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0895813Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0895818Z 
2022-11-23T02:20:42.0895889Z Running tests...
2022-11-23T02:20:42.0896071Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0896356Z   test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23113
2022-11-23T02:20:42.0896511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23114
2022-11-23T02:20:42.0896666Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23115
2022-11-23T02:20:42.0896820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23116
2022-11-23T02:20:42.0897096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0897218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0897500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0897633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0897808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikjqd5l_
2022-11-23T02:20:42.0898007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikjqd5l_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0898279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0898402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0898681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0898813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0898996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpep22dhjs
2022-11-23T02:20:42.0899194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpep22dhjs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0899454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0899579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0899858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0899994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0900175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccranglu
2022-11-23T02:20:42.0900370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccranglu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0900639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0900762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0901043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0901166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0901348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35kowxsb
2022-11-23T02:20:42.0901539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35kowxsb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0901729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0901892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0902052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0902209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0902273Z ok (2.675s)
2022-11-23T02:20:42.0902278Z 
2022-11-23T02:20:42.0902463Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0902563Z Ran 1 test in 2.676s
2022-11-23T02:20:42.0902569Z 
2022-11-23T02:20:42.0902626Z OK
2022-11-23T02:20:42.0902631Z 
2022-11-23T02:20:42.0902713Z Generating XML reports...
2022-11-23T02:20:42.0903094Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015320.xml
2022-11-23T02:20:42.0903366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0903488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0903766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0903898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0904068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ctvtd4h
2022-11-23T02:20:42.0904265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ctvtd4h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0904518Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0904524Z 
2022-11-23T02:20:42.0904594Z Running tests...
2022-11-23T02:20:42.0904784Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0905057Z   test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23356
2022-11-23T02:20:42.0905213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23357
2022-11-23T02:20:42.0905366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23358
2022-11-23T02:20:42.0905518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23359
2022-11-23T02:20:42.0905780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0905902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0906284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0906419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0906606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkezp050c
2022-11-23T02:20:42.0906801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkezp050c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0907074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0907199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0907468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0907602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0907783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yohvtjo
2022-11-23T02:20:42.0907980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yohvtjo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0908284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0908409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0908686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0908818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0909001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty6k86fe
2022-11-23T02:20:42.0909214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty6k86fe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0909483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0909602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0909884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0910016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0910199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkc5oau__
2022-11-23T02:20:42.0910392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkc5oau__/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0910555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0910708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0910870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0911031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0911098Z ok (2.578s)
2022-11-23T02:20:42.0911103Z 
2022-11-23T02:20:42.0911299Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0911372Z Ran 1 test in 2.578s
2022-11-23T02:20:42.0911377Z 
2022-11-23T02:20:42.0911435Z OK
2022-11-23T02:20:42.0911439Z 
2022-11-23T02:20:42.0911529Z Generating XML reports...
2022-11-23T02:20:42.0911907Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015324.xml
2022-11-23T02:20:42.0912166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0912288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0912571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0912702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0912882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5esnaee
2022-11-23T02:20:42.0913077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5esnaee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0913330Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0913334Z 
2022-11-23T02:20:42.0913403Z Running tests...
2022-11-23T02:20:42.0913583Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0913810Z   test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23589
2022-11-23T02:20:42.0913970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23590
2022-11-23T02:20:42.0914122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23591
2022-11-23T02:20:42.0914281Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23592
2022-11-23T02:20:42.0914580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0914704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0914986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0915120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0915290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwveh11s5
2022-11-23T02:20:42.0915482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwveh11s5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0915782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0915907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0916189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0916322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0916506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfib6djw
2022-11-23T02:20:42.0916700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfib6djw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0916971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0917081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0917360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0917493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0917673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbcafgb4
2022-11-23T02:20:42.0917867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbcafgb4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0918138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0918258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0918535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0918655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0918836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l2xonij
2022-11-23T02:20:42.0919029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l2xonij/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0919189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0919352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0919510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0919669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0919733Z ok (2.585s)
2022-11-23T02:20:42.0919738Z 
2022-11-23T02:20:42.0919933Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0919994Z Ran 1 test in 2.585s
2022-11-23T02:20:42.0919999Z 
2022-11-23T02:20:42.0920059Z OK
2022-11-23T02:20:42.0920064Z 
2022-11-23T02:20:42.0920148Z Generating XML reports...
2022-11-23T02:20:42.0920499Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015329.xml
2022-11-23T02:20:42.0920768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0920923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0921203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0921335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0921507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp80f21wz
2022-11-23T02:20:42.0921702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp80f21wz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0921954Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0921988Z 
2022-11-23T02:20:42.0922060Z Running tests...
2022-11-23T02:20:42.0922255Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0922499Z   test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23828
2022-11-23T02:20:42.0922658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23829
2022-11-23T02:20:42.0922810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23830
2022-11-23T02:20:42.0922959Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23831
2022-11-23T02:20:42.0923223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0923344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0923624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0923758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0923949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bawj28i
2022-11-23T02:20:42.0924144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bawj28i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0924413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0924533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0924801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0925036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0925222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcrianz1
2022-11-23T02:20:42.0925420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcrianz1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0925693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0925813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0926094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0926227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0926409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox_ebbjt
2022-11-23T02:20:42.0926590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox_ebbjt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0926860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0926983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0927260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0927393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0927611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0ct7j4u
2022-11-23T02:20:42.0927806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0ct7j4u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0927969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0928131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0928279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0928436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0928542Z ok (2.575s)
2022-11-23T02:20:42.0928547Z 
2022-11-23T02:20:42.0928743Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0928818Z Ran 1 test in 2.576s
2022-11-23T02:20:42.0928823Z 
2022-11-23T02:20:42.0928884Z OK
2022-11-23T02:20:42.0928888Z 
2022-11-23T02:20:42.0928972Z Generating XML reports...
2022-11-23T02:20:42.0929321Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015333.xml
2022-11-23T02:20:42.0929584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0929707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0929985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0930119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0930302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1aw2adt
2022-11-23T02:20:42.0930498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1aw2adt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0930753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0930758Z 
2022-11-23T02:20:42.0930828Z Running tests...
2022-11-23T02:20:42.0931023Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0931262Z   test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24055
2022-11-23T02:20:42.0931418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24056
2022-11-23T02:20:42.0931571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24057
2022-11-23T02:20:42.0931725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24058
2022-11-23T02:20:42.0931999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0932123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0932410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0932545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0932719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5001ba4k
2022-11-23T02:20:42.0932910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5001ba4k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0933184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0933307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0933587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0933718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0933937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0gh8sll
2022-11-23T02:20:42.0934131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0gh8sll/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0934401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0934512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0934790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0935039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0935275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5n0jj22
2022-11-23T02:20:42.0935470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5n0jj22/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0935745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0935870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0936152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0936273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0936456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_16qcgv
2022-11-23T02:20:42.0936647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_16qcgv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0936813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0936975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0937134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0937297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0937366Z ok (2.674s)
2022-11-23T02:20:42.0937371Z 
2022-11-23T02:20:42.0937570Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0937632Z Ran 1 test in 2.674s
2022-11-23T02:20:42.0937637Z 
2022-11-23T02:20:42.0937697Z OK
2022-11-23T02:20:42.0937702Z 
2022-11-23T02:20:42.0937785Z Generating XML reports...
2022-11-23T02:20:42.0938135Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015337.xml
2022-11-23T02:20:42.0938409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0938528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0938801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0938942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0939126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmcs0tt0
2022-11-23T02:20:42.0939322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmcs0tt0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0939577Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0939582Z 
2022-11-23T02:20:42.0939654Z Running tests...
2022-11-23T02:20:42.0939849Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0940093Z   test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24282
2022-11-23T02:20:42.0940248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24283
2022-11-23T02:20:42.0940388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24284
2022-11-23T02:20:42.0940609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24285
2022-11-23T02:20:42.0940882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0941005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0941287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0941420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0941632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphn0qrgf9
2022-11-23T02:20:42.0941827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphn0qrgf9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0942100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0942210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0942491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0942624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0942805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyqm7v1s
2022-11-23T02:20:42.0943000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyqm7v1s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0943276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0943402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0943682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0943805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0943988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8tp5txl
2022-11-23T02:20:42.0944179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8tp5txl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0944449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0944570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0944849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0944983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0945166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hhpkhft
2022-11-23T02:20:42.0945362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hhpkhft/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0945513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0945677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0945840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0946001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0946067Z ok (2.577s)
2022-11-23T02:20:42.0946073Z 
2022-11-23T02:20:42.0946267Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0946345Z Ran 1 test in 2.577s
2022-11-23T02:20:42.0946351Z 
2022-11-23T02:20:42.0946409Z OK
2022-11-23T02:20:42.0946413Z 
2022-11-23T02:20:42.0946482Z Generating XML reports...
2022-11-23T02:20:42.0946827Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015342.xml
2022-11-23T02:20:42.0947129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0947254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0947532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0947664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0947847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubn2ldrv
2022-11-23T02:20:42.0948069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubn2ldrv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0948323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0948328Z 
2022-11-23T02:20:42.0948387Z Running tests...
2022-11-23T02:20:42.0948580Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0948811Z   test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24509
2022-11-23T02:20:42.0948967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24510
2022-11-23T02:20:42.0949121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24511
2022-11-23T02:20:42.0949272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24512
2022-11-23T02:20:42.0949546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0949669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0949938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0950071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0950259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1s1p5so
2022-11-23T02:20:42.0950453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1s1p5so/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0950725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0950846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0951128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0951262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0951445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyj7nj08
2022-11-23T02:20:42.0951628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyj7nj08/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0951899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0952021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0952301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0952433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0952617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp656xt70h
2022-11-23T02:20:42.0952809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp656xt70h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0953083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0953204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0953497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0953631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0953814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeiu_yeu6
2022-11-23T02:20:42.0954007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeiu_yeu6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0954169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0954330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0954519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0954677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0954731Z ok (2.681s)
2022-11-23T02:20:42.0954736Z 
2022-11-23T02:20:42.0954931Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0955005Z Ran 1 test in 2.681s
2022-11-23T02:20:42.0955010Z 
2022-11-23T02:20:42.0955070Z OK
2022-11-23T02:20:42.0955075Z 
2022-11-23T02:20:42.0955158Z Generating XML reports...
2022-11-23T02:20:42.0955502Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015346.xml
2022-11-23T02:20:42.0955773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0955896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0956178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0956300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0956483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d87o0gm
2022-11-23T02:20:42.0956679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d87o0gm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0956933Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0956938Z 
2022-11-23T02:20:42.0957010Z Running tests...
2022-11-23T02:20:42.0957203Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0957441Z   test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24732
2022-11-23T02:20:42.0957599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24733
2022-11-23T02:20:42.0957743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24734
2022-11-23T02:20:42.0957896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24735
2022-11-23T02:20:42.0958170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0958292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0958573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0958708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0958892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wl2sp0r
2022-11-23T02:20:42.0959087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wl2sp0r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0959362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0959470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0959752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0959920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0960104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rvz4d5a
2022-11-23T02:20:42.0960298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rvz4d5a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0960568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0960690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0960967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0961113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0961293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g2zx_5n
2022-11-23T02:20:42.0961485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g2zx_5n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0961755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0961875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0962152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0962287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0962469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojmkm_34
2022-11-23T02:20:42.0962662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojmkm_34/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0962811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0962974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0963135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0963297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0963363Z ok (2.674s)
2022-11-23T02:20:42.0963368Z 
2022-11-23T02:20:42.0963563Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0963637Z Ran 1 test in 2.675s
2022-11-23T02:20:42.0963642Z 
2022-11-23T02:20:42.0963701Z OK
2022-11-23T02:20:42.0963706Z 
2022-11-23T02:20:42.0963775Z Generating XML reports...
2022-11-23T02:20:42.0964120Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015351.xml
2022-11-23T02:20:42.0964390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0964514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0964797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0965009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0965197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp394bmz1c
2022-11-23T02:20:42.0965389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp394bmz1c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0965643Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0965650Z 
2022-11-23T02:20:42.0965709Z Running tests...
2022-11-23T02:20:42.0965901Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0966129Z   test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24955
2022-11-23T02:20:42.0966318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24956
2022-11-23T02:20:42.0966473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24957
2022-11-23T02:20:42.0966624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24958
2022-11-23T02:20:42.0966900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0967021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0967288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0967448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0967632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgm6a73d
2022-11-23T02:20:42.0967827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgm6a73d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0968100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0968222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0968503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0968636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0968822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw83tkds_
2022-11-23T02:20:42.0969006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw83tkds_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0969277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0969398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0969681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0969814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0969998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbk5r5fy
2022-11-23T02:20:42.0970193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbk5r5fy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0970463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0970585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0970854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0970987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0971173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9mpmtft
2022-11-23T02:20:42.0971373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9mpmtft/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0971534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0971695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0971855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0972013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0972069Z ok (2.674s)
2022-11-23T02:20:42.0972075Z 
2022-11-23T02:20:42.0972269Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0972342Z Ran 1 test in 2.674s
2022-11-23T02:20:42.0972348Z 
2022-11-23T02:20:42.0972406Z OK
2022-11-23T02:20:42.0972411Z 
2022-11-23T02:20:42.0972491Z Generating XML reports...
2022-11-23T02:20:42.0972865Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015356.xml
2022-11-23T02:20:42.0973138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0973259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0973539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0973661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0973892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8geb4fn
2022-11-23T02:20:42.0974087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8geb4fn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0974341Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0974346Z 
2022-11-23T02:20:42.0974418Z Running tests...
2022-11-23T02:20:42.0974613Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0974972Z   test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25178
2022-11-23T02:20:42.0975128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25179
2022-11-23T02:20:42.0975268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25180
2022-11-23T02:20:42.0975422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25181
2022-11-23T02:20:42.0975700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0975821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0976104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0976241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0976427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6e2u2pn
2022-11-23T02:20:42.0976620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6e2u2pn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0976892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0977001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0977283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0977415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0977598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz9n4uoo
2022-11-23T02:20:42.0977795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz9n4uoo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0978067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0978189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0978468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0978588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0978772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcz1f9ua
2022-11-23T02:20:42.0978965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcz1f9ua/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0979236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0979405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0979684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0979816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0979998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwor48ase
2022-11-23T02:20:42.0980190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwor48ase/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0980340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0980540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0980701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0980861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0980929Z ok (2.476s)
2022-11-23T02:20:42.0980934Z 
2022-11-23T02:20:42.0981132Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0981207Z Ran 1 test in 2.476s
2022-11-23T02:20:42.0981212Z 
2022-11-23T02:20:42.0981271Z OK
2022-11-23T02:20:42.0981275Z 
2022-11-23T02:20:42.0981345Z Generating XML reports...
2022-11-23T02:20:42.0981693Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015400.xml
2022-11-23T02:20:42.0981966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0982091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0982368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0982500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0982685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_32q4sq
2022-11-23T02:20:42.0982877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_32q4sq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0983129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0983135Z 
2022-11-23T02:20:42.0983193Z Running tests...
2022-11-23T02:20:42.0983390Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0983630Z   test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25401
2022-11-23T02:20:42.0983790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25402
2022-11-23T02:20:42.0983944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25403
2022-11-23T02:20:42.0984097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25404
2022-11-23T02:20:42.0984370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0984491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0984772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0984894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0985075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sy0mfwb
2022-11-23T02:20:42.0985271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sy0mfwb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0985540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0985662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0985975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0986108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0986290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lpddrz0
2022-11-23T02:20:42.0986471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lpddrz0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0986743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0986891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0987172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0987305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0987488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0v98uyen
2022-11-23T02:20:42.0987678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0v98uyen/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0987946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0988067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0988335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0988469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0988650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_44bgmf
2022-11-23T02:20:42.0988842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_44bgmf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0989006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0989168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0989327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0989487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0989541Z ok (2.475s)
2022-11-23T02:20:42.0989547Z 
2022-11-23T02:20:42.0989744Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0989819Z Ran 1 test in 2.475s
2022-11-23T02:20:42.0989826Z 
2022-11-23T02:20:42.0989885Z OK
2022-11-23T02:20:42.0989890Z 
2022-11-23T02:20:42.0989971Z Generating XML reports...
2022-11-23T02:20:42.0990322Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015404.xml
2022-11-23T02:20:42.0990596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0990718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0990998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0991118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0991301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfx1et64
2022-11-23T02:20:42.0991493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfx1et64/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0991747Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.0991752Z 
2022-11-23T02:20:42.0991825Z Running tests...
2022-11-23T02:20:42.0992023Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0992301Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25624
2022-11-23T02:20:42.0992459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25625
2022-11-23T02:20:42.0992599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25626
2022-11-23T02:20:42.0992753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25627
2022-11-23T02:20:42.0993024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0993177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0993459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0993593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0993779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzmihy24
2022-11-23T02:20:42.0993974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzmihy24/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0994244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0994354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0994633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0994766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0994954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw66b0wd_
2022-11-23T02:20:42.0995146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw66b0wd_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0995418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0995540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0995818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0995950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0996120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5w3e5ip
2022-11-23T02:20:42.0996312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5w3e5ip/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0996585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0996704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0996982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0997115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0997297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuuj5i2t
2022-11-23T02:20:42.0997488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuuj5i2t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.0997637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.0997801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.0997961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.0998125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.0998192Z ok (2.481s)
2022-11-23T02:20:42.0998198Z 
2022-11-23T02:20:42.0998394Z ----------------------------------------------------------------------
2022-11-23T02:20:42.0998467Z Ran 1 test in 2.481s
2022-11-23T02:20:42.0998500Z 
2022-11-23T02:20:42.0998561Z OK
2022-11-23T02:20:42.0998566Z 
2022-11-23T02:20:42.0998635Z Generating XML reports...
2022-11-23T02:20:42.0998983Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015409.xml
2022-11-23T02:20:42.0999255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.0999376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.0999657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.0999818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.0999999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyiihn1x4
2022-11-23T02:20:42.1000195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyiihn1x4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1000451Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1000456Z 
2022-11-23T02:20:42.1000513Z Running tests...
2022-11-23T02:20:42.1000708Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1000931Z   test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25847
2022-11-23T02:20:42.1001090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25848
2022-11-23T02:20:42.1001245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25849
2022-11-23T02:20:42.1001397Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25850
2022-11-23T02:20:42.1001670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1001793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1002077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1002198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1002383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbfndct2
2022-11-23T02:20:42.1002579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbfndct2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1002852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1002977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1003256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1003390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1003577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi1xo7je
2022-11-23T02:20:42.1003760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi1xo7je/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1004029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1004150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1004426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1004560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1004741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1oy_siel
2022-11-23T02:20:42.1005011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1oy_siel/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1005325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1005450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1005715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1005849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1006032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmu2wsv4
2022-11-23T02:20:42.1006255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmu2wsv4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1006418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1006580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1006743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1006903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1006955Z ok (2.774s)
2022-11-23T02:20:42.1006973Z 
2022-11-23T02:20:42.1007158Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1007231Z Ran 1 test in 2.774s
2022-11-23T02:20:42.1007236Z 
2022-11-23T02:20:42.1007296Z OK
2022-11-23T02:20:42.1007301Z 
2022-11-23T02:20:42.1007382Z Generating XML reports...
2022-11-23T02:20:42.1007730Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015413.xml
2022-11-23T02:20:42.1008002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1008123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1008404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1008524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1008708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagb7qywg
2022-11-23T02:20:42.1008902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagb7qywg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1009159Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1009164Z 
2022-11-23T02:20:42.1009235Z Running tests...
2022-11-23T02:20:42.1009432Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1009654Z   test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26086
2022-11-23T02:20:42.1009813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26087
2022-11-23T02:20:42.1009955Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26088
2022-11-23T02:20:42.1010105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26089
2022-11-23T02:20:42.1010376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1010498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1010777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1010911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1011096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopxs4ocb
2022-11-23T02:20:42.1011290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopxs4ocb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1011590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1011701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1011980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1012113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1012302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk14mxhsr
2022-11-23T02:20:42.1012495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk14mxhsr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1012791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1012912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1013193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1013325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1013497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68xufhm_
2022-11-23T02:20:42.1013688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68xufhm_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1013958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1014078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1014359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1014490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1014672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip9dmcm1
2022-11-23T02:20:42.1014980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip9dmcm1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1015132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1015296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1015456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1015615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1015681Z ok (2.677s)
2022-11-23T02:20:42.1015689Z 
2022-11-23T02:20:42.1015886Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1015960Z Ran 1 test in 2.677s
2022-11-23T02:20:42.1015965Z 
2022-11-23T02:20:42.1016023Z OK
2022-11-23T02:20:42.1016028Z 
2022-11-23T02:20:42.1016098Z Generating XML reports...
2022-11-23T02:20:42.1016450Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015418.xml
2022-11-23T02:20:42.1016722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1016844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1017123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1017257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1017441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx6612mk
2022-11-23T02:20:42.1017637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx6612mk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1017891Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1017896Z 
2022-11-23T02:20:42.1017953Z Running tests...
2022-11-23T02:20:42.1018196Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1018441Z   test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26325
2022-11-23T02:20:42.1018596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26326
2022-11-23T02:20:42.1018749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26327
2022-11-23T02:20:42.1018900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26328
2022-11-23T02:20:42.1019220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1019341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1019621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1019745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1019930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyy0qati
2022-11-23T02:20:42.1020126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyy0qati/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1020399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1020519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1020798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1020933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1021115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapzgaceh
2022-11-23T02:20:42.1021301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapzgaceh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1021570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1021689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1021968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1022100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1022282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp450pmypn
2022-11-23T02:20:42.1022478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp450pmypn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1022748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1022869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1023136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1023269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1023450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh28nmkk
2022-11-23T02:20:42.1023643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh28nmkk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1023809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1023974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1024132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1024290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1024344Z ok (2.274s)
2022-11-23T02:20:42.1024361Z 
2022-11-23T02:20:42.1024568Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1024644Z Ran 1 test in 2.274s
2022-11-23T02:20:42.1024649Z 
2022-11-23T02:20:42.1024709Z OK
2022-11-23T02:20:42.1024713Z 
2022-11-23T02:20:42.1024795Z Generating XML reports...
2022-11-23T02:20:42.1025142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015422.xml
2022-11-23T02:20:42.1025412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1025561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1025840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1025962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1026147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d7gxvq1
2022-11-23T02:20:42.1026342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d7gxvq1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1026594Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1026599Z 
2022-11-23T02:20:42.1026672Z Running tests...
2022-11-23T02:20:42.1026866Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1027133Z   test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26396
2022-11-23T02:20:42.1027290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26397
2022-11-23T02:20:42.1027442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26398
2022-11-23T02:20:42.1027582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26399
2022-11-23T02:20:42.1027859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1027980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1028260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1028393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1028575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__we3k4l
2022-11-23T02:20:42.1028771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__we3k4l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1029042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1029152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1029431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1029564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1029748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfa2w3vi
2022-11-23T02:20:42.1029941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfa2w3vi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1030211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1030334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1030610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1030742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1030943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnfq5nn4
2022-11-23T02:20:42.1031136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnfq5nn4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1031406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1031526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1031805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1031938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1032149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcq1gszr
2022-11-23T02:20:42.1032339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcq1gszr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1032490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1032654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1032816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1032976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1033042Z ok (2.578s)
2022-11-23T02:20:42.1033047Z 
2022-11-23T02:20:42.1033242Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1033314Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1033319Z 
2022-11-23T02:20:42.1033382Z OK
2022-11-23T02:20:42.1033387Z 
2022-11-23T02:20:42.1033468Z Generating XML reports...
2022-11-23T02:20:42.1033802Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015426.xml
2022-11-23T02:20:42.1034074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1034200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1034477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1034609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1034791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29l8hsvl
2022-11-23T02:20:42.1034984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29l8hsvl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1035237Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1035242Z 
2022-11-23T02:20:42.1035311Z Running tests...
2022-11-23T02:20:42.1035494Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1035724Z   test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26623
2022-11-23T02:20:42.1035881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26624
2022-11-23T02:20:42.1036035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26625
2022-11-23T02:20:42.1036187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26626
2022-11-23T02:20:42.1036460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1036581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1036867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1036988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1037170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfq2ijwnq
2022-11-23T02:20:42.1037401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfq2ijwnq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1037674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1037797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1038077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1038210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1038459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ktwsg7x
2022-11-23T02:20:42.1038639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ktwsg7x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1038910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1039033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1039310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1039446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1039628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f8cpbku
2022-11-23T02:20:42.1039821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f8cpbku/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1040091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1040214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1040478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1040615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1040795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ht12tfu
2022-11-23T02:20:42.1040988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ht12tfu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1041151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1041313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1041472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1041634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1041688Z ok (2.576s)
2022-11-23T02:20:42.1041707Z 
2022-11-23T02:20:42.1041891Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1041964Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1041969Z 
2022-11-23T02:20:42.1042028Z OK
2022-11-23T02:20:42.1042035Z 
2022-11-23T02:20:42.1042119Z Generating XML reports...
2022-11-23T02:20:42.1042468Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015431.xml
2022-11-23T02:20:42.1042742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1042863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1043141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1043263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1043447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmytz5e2
2022-11-23T02:20:42.1043641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmytz5e2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1043925Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1043931Z 
2022-11-23T02:20:42.1044004Z Running tests...
2022-11-23T02:20:42.1044198Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1044464Z   test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26850
2022-11-23T02:20:42.1044619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26851
2022-11-23T02:20:42.1044800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26852
2022-11-23T02:20:42.1045006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26853
2022-11-23T02:20:42.1045287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1045410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1045691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1045824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1046007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyx0s96o
2022-11-23T02:20:42.1046202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyx0s96o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1046471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1046582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1046861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1046993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1047177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i3ap3bs
2022-11-23T02:20:42.1047369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i3ap3bs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1047638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1047758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1048038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1048173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1048342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1mhikjd
2022-11-23T02:20:42.1048536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1mhikjd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1048807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1048927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1049203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1049336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1049518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd8r3mja
2022-11-23T02:20:42.1049707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd8r3mja/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1049872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1050022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1050231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1050394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1050462Z ok (2.575s)
2022-11-23T02:20:42.1050467Z 
2022-11-23T02:20:42.1050664Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1050738Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1050744Z 
2022-11-23T02:20:42.1050802Z OK
2022-11-23T02:20:42.1050807Z 
2022-11-23T02:20:42.1050888Z Generating XML reports...
2022-11-23T02:20:42.1051222Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015435.xml
2022-11-23T02:20:42.1051525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1051647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1051929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1052062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1052246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvwzftlk
2022-11-23T02:20:42.1052444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvwzftlk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1052699Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1052704Z 
2022-11-23T02:20:42.1052775Z Running tests...
2022-11-23T02:20:42.1052956Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1053216Z   test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27077
2022-11-23T02:20:42.1053375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27078
2022-11-23T02:20:42.1053530Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27079
2022-11-23T02:20:42.1053679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27080
2022-11-23T02:20:42.1053950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1054071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1054349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1054473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1054656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmnr6dptf
2022-11-23T02:20:42.1054959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmnr6dptf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1055236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1055358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1055636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1055768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1055950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34ltuz45
2022-11-23T02:20:42.1056141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34ltuz45/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1056400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1056519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1056841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1056977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1057159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwwqaa0e
2022-11-23T02:20:42.1057351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwwqaa0e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1057618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1057738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1058040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1058173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1058354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zzlaq9l
2022-11-23T02:20:42.1058549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zzlaq9l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1058712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1058874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1059035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1059196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1059260Z ok (2.679s)
2022-11-23T02:20:42.1059268Z 
2022-11-23T02:20:42.1059452Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1059525Z Ran 1 test in 2.679s
2022-11-23T02:20:42.1059530Z 
2022-11-23T02:20:42.1059588Z OK
2022-11-23T02:20:42.1059593Z 
2022-11-23T02:20:42.1059676Z Generating XML reports...
2022-11-23T02:20:42.1060021Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015440.xml
2022-11-23T02:20:42.1060291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1060414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1060693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1060813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1060995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunwu58x1
2022-11-23T02:20:42.1061188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunwu58x1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1061439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1061444Z 
2022-11-23T02:20:42.1061517Z Running tests...
2022-11-23T02:20:42.1061712Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1061936Z   test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27304
2022-11-23T02:20:42.1062092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27305
2022-11-23T02:20:42.1062244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27306
2022-11-23T02:20:42.1062383Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27307
2022-11-23T02:20:42.1062656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1062776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1063056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1063221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1063404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6d_e8x_
2022-11-23T02:20:42.1063595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6d_e8x_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1063863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1063985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1064249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1064424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1064609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4inwjgk
2022-11-23T02:20:42.1064805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4inwjgk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1065073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1065192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1065471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1065602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1065772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjykaxecz
2022-11-23T02:20:42.1065966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjykaxecz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1066233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1066356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1066632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1066765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1066947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrpt3wju
2022-11-23T02:20:42.1067138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrpt3wju/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1067299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1067450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1067609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1067770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1067836Z ok (2.580s)
2022-11-23T02:20:42.1067841Z 
2022-11-23T02:20:42.1068037Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1068111Z Ran 1 test in 2.580s
2022-11-23T02:20:42.1068116Z 
2022-11-23T02:20:42.1068174Z OK
2022-11-23T02:20:42.1068178Z 
2022-11-23T02:20:42.1068260Z Generating XML reports...
2022-11-23T02:20:42.1068592Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015444.xml
2022-11-23T02:20:42.1068863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1068986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1069265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1069399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1069614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifq8q9i_
2022-11-23T02:20:42.1069808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifq8q9i_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1070060Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1070065Z 
2022-11-23T02:20:42.1070135Z Running tests...
2022-11-23T02:20:42.1070316Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1070541Z   test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27527
2022-11-23T02:20:42.1070727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27528
2022-11-23T02:20:42.1070880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27529
2022-11-23T02:20:42.1071033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27530
2022-11-23T02:20:42.1071308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1071430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1071710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1071829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1072009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z8yxrfm
2022-11-23T02:20:42.1072200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z8yxrfm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1072472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1072592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1072871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1073003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1073185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp727611by
2022-11-23T02:20:42.1073374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp727611by/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1073629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1073750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1074027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1074162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1074343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkq75lsv
2022-11-23T02:20:42.1074536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkq75lsv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1074804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1074924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1075190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1075322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1075504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpag3w7yyg
2022-11-23T02:20:42.1075701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpag3w7yyg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1075861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1076051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1076212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1076372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1076437Z ok (2.677s)
2022-11-23T02:20:42.1076442Z 
2022-11-23T02:20:42.1076625Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1076699Z Ran 1 test in 2.678s
2022-11-23T02:20:42.1076704Z 
2022-11-23T02:20:42.1076762Z OK
2022-11-23T02:20:42.1076795Z 
2022-11-23T02:20:42.1076880Z Generating XML reports...
2022-11-23T02:20:42.1077227Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015449.xml
2022-11-23T02:20:42.1077499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1077624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1077906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1078027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1078210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq5g4zhi
2022-11-23T02:20:42.1078401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq5g4zhi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1078655Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1078662Z 
2022-11-23T02:20:42.1078734Z Running tests...
2022-11-23T02:20:42.1078929Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1079163Z   test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27766
2022-11-23T02:20:42.1079318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27767
2022-11-23T02:20:42.1079471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27768
2022-11-23T02:20:42.1079610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27769
2022-11-23T02:20:42.1079886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1080007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1080289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1080421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1080606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1nr79_h
2022-11-23T02:20:42.1080802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1nr79_h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1081071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1081191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1081455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1081588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1081767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k9s9gcn
2022-11-23T02:20:42.1081962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k9s9gcn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1082232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1082384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1082663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1082796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1082964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81y5boz_
2022-11-23T02:20:42.1083235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1083355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1083664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1083796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1083978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp423gcpbr
2022-11-23T02:20:42.1084171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81y5boz_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1084360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp423gcpbr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1084522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1084673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1084836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1085081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1085151Z ok (2.576s)
2022-11-23T02:20:42.1085157Z 
2022-11-23T02:20:42.1085354Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1085426Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1085431Z 
2022-11-23T02:20:42.1085491Z OK
2022-11-23T02:20:42.1085495Z 
2022-11-23T02:20:42.1085579Z Generating XML reports...
2022-11-23T02:20:42.1085915Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015453.xml
2022-11-23T02:20:42.1086185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1086308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1086585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1086719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1086901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizz7mh3o
2022-11-23T02:20:42.1087095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizz7mh3o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1087349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1087355Z 
2022-11-23T02:20:42.1087425Z Running tests...
2022-11-23T02:20:42.1087608Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1087853Z   test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28005
2022-11-23T02:20:42.1088008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28006
2022-11-23T02:20:42.1088160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28007
2022-11-23T02:20:42.1088314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28008
2022-11-23T02:20:42.1088589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1088709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1089028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1089153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1089337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wyhjjsc
2022-11-23T02:20:42.1089532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wyhjjsc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1089805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1089957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1090237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1090369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1090557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprctm7v04
2022-11-23T02:20:42.1090753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprctm7v04/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1091012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1091132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1091410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1091543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1091727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyrgj676
2022-11-23T02:20:42.1091921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyrgj676/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1092192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1092315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1092593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1092713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1092899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6injy6d
2022-11-23T02:20:42.1093094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6injy6d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1093259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1093421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1093581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1093741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1093807Z ok (2.575s)
2022-11-23T02:20:42.1093811Z 
2022-11-23T02:20:42.1093995Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1094069Z Ran 1 test in 2.575s
2022-11-23T02:20:42.1094074Z 
2022-11-23T02:20:42.1094134Z OK
2022-11-23T02:20:42.1094139Z 
2022-11-23T02:20:42.1094223Z Generating XML reports...
2022-11-23T02:20:42.1094571Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015458.xml
2022-11-23T02:20:42.1094952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1095076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1095358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1095544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1095719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwb1tcwjd
2022-11-23T02:20:42.1095914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwb1tcwjd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1096168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1096173Z 
2022-11-23T02:20:42.1096244Z Running tests...
2022-11-23T02:20:42.1096440Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1096722Z   test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28228
2022-11-23T02:20:42.1096878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28229
2022-11-23T02:20:42.1097034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28230
2022-11-23T02:20:42.1097175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28231
2022-11-23T02:20:42.1097449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1097573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1097854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1097989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1098175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp400srqs0
2022-11-23T02:20:42.1098369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp400srqs0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1098640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1098764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1099032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1099166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1099347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3i814n0
2022-11-23T02:20:42.1099538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3i814n0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1099813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1099935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1100212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1100347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1100518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztdpcgdm
2022-11-23T02:20:42.1100713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztdpcgdm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1100982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1101104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1101385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1101519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1101704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpfv7__p
2022-11-23T02:20:42.1101929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpfv7__p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1102092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1102243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1102404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1102564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1102631Z ok (2.585s)
2022-11-23T02:20:42.1102636Z 
2022-11-23T02:20:42.1102832Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1102936Z Ran 1 test in 2.585s
2022-11-23T02:20:42.1102941Z 
2022-11-23T02:20:42.1103001Z OK
2022-11-23T02:20:42.1103006Z 
2022-11-23T02:20:42.1103087Z Generating XML reports...
2022-11-23T02:20:42.1103424Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015502.xml
2022-11-23T02:20:42.1103700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1103821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1104101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1104237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1104422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitnfdh70
2022-11-23T02:20:42.1104621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitnfdh70/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1104875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1104880Z 
2022-11-23T02:20:42.1104951Z Running tests...
2022-11-23T02:20:42.1105134Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1105373Z   test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28451
2022-11-23T02:20:42.1105528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28452
2022-11-23T02:20:42.1105681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28453
2022-11-23T02:20:42.1105831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28454
2022-11-23T02:20:42.1106101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1106224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1106500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1106634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1106810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2fewv2d
2022-11-23T02:20:42.1107005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2fewv2d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1107275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1107400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1107685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1107820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1108004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsaxkk3lh
2022-11-23T02:20:42.1108200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsaxkk3lh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1108487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1108609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1108885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1109018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1109201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vadjtcp
2022-11-23T02:20:42.1109395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vadjtcp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1109709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1109831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1110109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1110231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1110412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd_kteqx
2022-11-23T02:20:42.1110603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd_kteqx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1110766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1110926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1111088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1111246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1111311Z ok (2.574s)
2022-11-23T02:20:42.1111317Z 
2022-11-23T02:20:42.1111502Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1111577Z Ran 1 test in 2.574s
2022-11-23T02:20:42.1111581Z 
2022-11-23T02:20:42.1111640Z OK
2022-11-23T02:20:42.1111644Z 
2022-11-23T02:20:42.1111728Z Generating XML reports...
2022-11-23T02:20:42.1112072Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015507.xml
2022-11-23T02:20:42.1112344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1112465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1112747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1112880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1113052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibqlt6se
2022-11-23T02:20:42.1113248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibqlt6se/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1113503Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1113508Z 
2022-11-23T02:20:42.1113579Z Running tests...
2022-11-23T02:20:42.1113775Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1114495Z   test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84046 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T02:20:42.1114504Z 
2022-11-23T02:20:42.1114696Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1114769Z Ran 1 test in 0.255s
2022-11-23T02:20:42.1114804Z 
2022-11-23T02:20:42.1114875Z OK (skipped=1)
2022-11-23T02:20:42.1114880Z 
2022-11-23T02:20:42.1114962Z Generating XML reports...
2022-11-23T02:20:42.1115296Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015511.xml
2022-11-23T02:20:42.1115570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1115695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1115978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1116142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1116326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp713d5ei3
2022-11-23T02:20:42.1116520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp713d5ei3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1116774Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1116779Z 
2022-11-23T02:20:42.1116837Z Running tests...
2022-11-23T02:20:42.1117033Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1117263Z   test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28684
2022-11-23T02:20:42.1117419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28685
2022-11-23T02:20:42.1117576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28686
2022-11-23T02:20:42.1117727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28687
2022-11-23T02:20:42.1118000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1118123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1118404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1118526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1118710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3o6pymcp
2022-11-23T02:20:42.1118906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3o6pymcp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1119178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1119302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1119582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1119718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1119900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuq318l1
2022-11-23T02:20:42.1120080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuq318l1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1120349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1120471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1120749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1120884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1121067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29vf0f1i
2022-11-23T02:20:42.1121259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29vf0f1i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1121561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1121684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1121951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1122083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1122266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8bayjcb
2022-11-23T02:20:42.1122495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8bayjcb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1122660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1122823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1122986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1123145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1123198Z ok (2.576s)
2022-11-23T02:20:42.1123215Z 
2022-11-23T02:20:42.1123400Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1123473Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1123478Z 
2022-11-23T02:20:42.1123537Z OK
2022-11-23T02:20:42.1123542Z 
2022-11-23T02:20:42.1123624Z Generating XML reports...
2022-11-23T02:20:42.1123972Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015513.xml
2022-11-23T02:20:42.1124246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1124368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1124647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1124769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1125013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67ntyvuk
2022-11-23T02:20:42.1125212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67ntyvuk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1125467Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1125472Z 
2022-11-23T02:20:42.1125543Z Running tests...
2022-11-23T02:20:42.1125740Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1125967Z   test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28911
2022-11-23T02:20:42.1126121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28912
2022-11-23T02:20:42.1126277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28913
2022-11-23T02:20:42.1126415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28914
2022-11-23T02:20:42.1126689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1126810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1127092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1127228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1127412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hlca72d
2022-11-23T02:20:42.1127607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hlca72d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1127912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1128021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1128301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1128433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1128615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mxxeuat
2022-11-23T02:20:42.1128811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mxxeuat/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1129112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1129232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1129512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1129643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1129813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_zcarva
2022-11-23T02:20:42.1130006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_zcarva/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1130275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1130397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1130678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1130811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1130993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitkc5o0o
2022-11-23T02:20:42.1131187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitkc5o0o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1131337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1131500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1131661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1131820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1131886Z ok (2.583s)
2022-11-23T02:20:42.1131893Z 
2022-11-23T02:20:42.1132089Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1132162Z Ran 1 test in 2.583s
2022-11-23T02:20:42.1132167Z 
2022-11-23T02:20:42.1132227Z OK
2022-11-23T02:20:42.1132231Z 
2022-11-23T02:20:42.1132312Z Generating XML reports...
2022-11-23T02:20:42.1132651Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015518.xml
2022-11-23T02:20:42.1132921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1133043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1133322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1133455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1133638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporzvmbwd
2022-11-23T02:20:42.1133837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporzvmbwd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1134089Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1134094Z 
2022-11-23T02:20:42.1134164Z Running tests...
2022-11-23T02:20:42.1134378Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1134626Z   test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29138
2022-11-23T02:20:42.1134780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29139
2022-11-23T02:20:42.1135045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29140
2022-11-23T02:20:42.1135196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29141
2022-11-23T02:20:42.1135519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1135643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1135924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1136046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1136228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qxgs6yr
2022-11-23T02:20:42.1136421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qxgs6yr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1136696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1136817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1137098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1137235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1137419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz22zp62
2022-11-23T02:20:42.1137615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz22zp62/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1137874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1137996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1138272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1138403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1138585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3zvxjtd
2022-11-23T02:20:42.1138780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3zvxjtd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1139053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1139173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1139440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1139573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1139753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xln69bi
2022-11-23T02:20:42.1139947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xln69bi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1140109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1140272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1140433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1140593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1140647Z ok (2.476s)
2022-11-23T02:20:42.1140664Z 
2022-11-23T02:20:42.1140918Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1140992Z Ran 1 test in 2.476s
2022-11-23T02:20:42.1140997Z 
2022-11-23T02:20:42.1141056Z OK
2022-11-23T02:20:42.1141061Z 
2022-11-23T02:20:42.1141143Z Generating XML reports...
2022-11-23T02:20:42.1141493Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015522.xml
2022-11-23T02:20:42.1141766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1141915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1142194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1142314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1142498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeka7imk
2022-11-23T02:20:42.1142691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeka7imk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1142945Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1142950Z 
2022-11-23T02:20:42.1143021Z Running tests...
2022-11-23T02:20:42.1143215Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1143460Z   test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29361
2022-11-23T02:20:42.1143618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29362
2022-11-23T02:20:42.1143774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29363
2022-11-23T02:20:42.1143913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29364
2022-11-23T02:20:42.1144187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1144310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1144590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1144724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1144908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ghd7osa
2022-11-23T02:20:42.1145105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ghd7osa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1145376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1145486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1145769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1145902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1146085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0zcxsvd
2022-11-23T02:20:42.1146279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0zcxsvd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1146551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1146678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1146955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1147088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1147291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdcczan8
2022-11-23T02:20:42.1147488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdcczan8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1147761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1147882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1148159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1148290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1148500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvv1nsyv
2022-11-23T02:20:42.1148694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvv1nsyv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1148844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1149010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1149170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1149331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1149397Z ok (2.575s)
2022-11-23T02:20:42.1149402Z 
2022-11-23T02:20:42.1149597Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1149670Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1149675Z 
2022-11-23T02:20:42.1149736Z OK
2022-11-23T02:20:42.1149741Z 
2022-11-23T02:20:42.1149822Z Generating XML reports...
2022-11-23T02:20:42.1150158Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015526.xml
2022-11-23T02:20:42.1150435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1150559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1150842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1150978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1151161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyuazx8q
2022-11-23T02:20:42.1151356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyuazx8q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1151612Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1151617Z 
2022-11-23T02:20:42.1151687Z Running tests...
2022-11-23T02:20:42.1151870Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1152100Z   test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29584
2022-11-23T02:20:42.1152255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29585
2022-11-23T02:20:42.1152410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29586
2022-11-23T02:20:42.1152560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29587
2022-11-23T02:20:42.1152833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1152955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1153238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1153359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1153541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x2vipda
2022-11-23T02:20:42.1153762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x2vipda/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1154041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1154163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1154442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1154575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1154799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjqq064p
2022-11-23T02:20:42.1154992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjqq064p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1155251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1155374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1155655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1155789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1155971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohp63xkh
2022-11-23T02:20:42.1156164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohp63xkh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1156436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1156560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1156827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1156959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1157142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplez35_nn
2022-11-23T02:20:42.1157333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplez35_nn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1157497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1157658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1157819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1157983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1158049Z ok (2.576s)
2022-11-23T02:20:42.1158054Z 
2022-11-23T02:20:42.1158238Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1158310Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1158315Z 
2022-11-23T02:20:42.1158374Z OK
2022-11-23T02:20:42.1158381Z 
2022-11-23T02:20:42.1158463Z Generating XML reports...
2022-11-23T02:20:42.1158809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015531.xml
2022-11-23T02:20:42.1159079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1159202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1159481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1159603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1159787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg5ogb14
2022-11-23T02:20:42.1159981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg5ogb14/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1160265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1160271Z 
2022-11-23T02:20:42.1160343Z Running tests...
2022-11-23T02:20:42.1160537Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1160782Z   test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29807
2022-11-23T02:20:42.1160938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29808
2022-11-23T02:20:42.1161091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29809
2022-11-23T02:20:42.1161259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29810
2022-11-23T02:20:42.1161534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1161654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1161937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1162071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1162255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnz3jx66
2022-11-23T02:20:42.1162449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnz3jx66/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1162720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1162832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1163110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1163244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1163429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvadl42wy
2022-11-23T02:20:42.1163623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvadl42wy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1163892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1164015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1164295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1164428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1164598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmaw1knx2
2022-11-23T02:20:42.1164791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmaw1knx2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1165139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1165262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1165547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1165679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1165862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgguvfchn
2022-11-23T02:20:42.1166056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgguvfchn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1166221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1166367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1166527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1166724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1166792Z ok (2.578s)
2022-11-23T02:20:42.1166798Z 
2022-11-23T02:20:42.1166995Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1167068Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1167073Z 
2022-11-23T02:20:42.1167132Z OK
2022-11-23T02:20:42.1167137Z 
2022-11-23T02:20:42.1167218Z Generating XML reports...
2022-11-23T02:20:42.1167550Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015535.xml
2022-11-23T02:20:42.1167853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1167974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1168254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1168388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1168572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8to_d5c
2022-11-23T02:20:42.1168764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8to_d5c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1169017Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1169023Z 
2022-11-23T02:20:42.1169093Z Running tests...
2022-11-23T02:20:42.1169275Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1169511Z   test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30030
2022-11-23T02:20:42.1169668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30031
2022-11-23T02:20:42.1169821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30032
2022-11-23T02:20:42.1169973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30033
2022-11-23T02:20:42.1170245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1170366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1170645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1170766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1170951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuer9xo72
2022-11-23T02:20:42.1171143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuer9xo72/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1171418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1171542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1171819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1171951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1172134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht854615
2022-11-23T02:20:42.1172322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht854615/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1172583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1172704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1172982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1173143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1173325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua9_83kg
2022-11-23T02:20:42.1173515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua9_83kg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1173785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1173905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1174172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1174332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1174513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8go1wqk
2022-11-23T02:20:42.1174708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8go1wqk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1174982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1175149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1175311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1175472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1175538Z ok (2.578s)
2022-11-23T02:20:42.1175543Z 
2022-11-23T02:20:42.1175730Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1175803Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1175808Z 
2022-11-23T02:20:42.1175866Z OK
2022-11-23T02:20:42.1175871Z 
2022-11-23T02:20:42.1175954Z Generating XML reports...
2022-11-23T02:20:42.1176304Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015540.xml
2022-11-23T02:20:42.1176577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1176698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1176976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1177097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1177279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wx63fcq
2022-11-23T02:20:42.1177473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wx63fcq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1177725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1177730Z 
2022-11-23T02:20:42.1177801Z Running tests...
2022-11-23T02:20:42.1177995Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1178242Z   test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30253
2022-11-23T02:20:42.1178399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30254
2022-11-23T02:20:42.1178552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30255
2022-11-23T02:20:42.1178692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30256
2022-11-23T02:20:42.1178965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1179090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1179372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1179551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1179737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigruvzdv
2022-11-23T02:20:42.1179933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigruvzdv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1180206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1180326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1180594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1180764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1180947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ec6v9yl
2022-11-23T02:20:42.1181140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ec6v9yl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1181413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1181536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1181813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1181947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1182117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqzlxi6m
2022-11-23T02:20:42.1182313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqzlxi6m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1182582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1182705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1182985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1183118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1183301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj0pfkhj
2022-11-23T02:20:42.1183495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj0pfkhj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1183659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1183808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1183971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1184130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1184198Z ok (2.676s)
2022-11-23T02:20:42.1184203Z 
2022-11-23T02:20:42.1184402Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1184477Z Ran 1 test in 2.676s
2022-11-23T02:20:42.1184482Z 
2022-11-23T02:20:42.1184540Z OK
2022-11-23T02:20:42.1184545Z 
2022-11-23T02:20:42.1184627Z Generating XML reports...
2022-11-23T02:20:42.1184962Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015544.xml
2022-11-23T02:20:42.1185233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1185355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1185635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1185768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1185950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvifoecf
2022-11-23T02:20:42.1186172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvifoecf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1186426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1186431Z 
2022-11-23T02:20:42.1186517Z Running tests...
2022-11-23T02:20:42.1186700Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1186950Z   test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30480
2022-11-23T02:20:42.1187134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30481
2022-11-23T02:20:42.1187288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30482
2022-11-23T02:20:42.1187439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30483
2022-11-23T02:20:42.1187713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1187834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1188115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1188235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1188419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_mdjy2x
2022-11-23T02:20:42.1188611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_mdjy2x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1188883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1189006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1189289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1189421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1189603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1_5zi8u
2022-11-23T02:20:42.1189793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1_5zi8u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1190051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1190171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1190451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1190585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1190766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk437b2jm
2022-11-23T02:20:42.1190960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk437b2jm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1191227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1191348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1191623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1191744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1191926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdzrjbud
2022-11-23T02:20:42.1192120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdzrjbud/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1192282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1192472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1192634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1192794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1192859Z ok (3.178s)
2022-11-23T02:20:42.1192864Z 
2022-11-23T02:20:42.1193049Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1193122Z Ran 1 test in 3.178s
2022-11-23T02:20:42.1193127Z 
2022-11-23T02:20:42.1193185Z OK
2022-11-23T02:20:42.1193216Z 
2022-11-23T02:20:42.1193301Z Generating XML reports...
2022-11-23T02:20:42.1193647Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015549.xml
2022-11-23T02:20:42.1193918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1194042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1194319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1194451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1194622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrqb_9xq
2022-11-23T02:20:42.1194816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrqb_9xq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1195068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1195075Z 
2022-11-23T02:20:42.1195147Z Running tests...
2022-11-23T02:20:42.1195341Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1195586Z   test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30535
2022-11-23T02:20:42.1195741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30536
2022-11-23T02:20:42.1195895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30537
2022-11-23T02:20:42.1196034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30538
2022-11-23T02:20:42.1196308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1196431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1196716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1196849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1197032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkza0_82g
2022-11-23T02:20:42.1197227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkza0_82g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1197499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1197621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1197888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1198023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1198202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05_a4d7c
2022-11-23T02:20:42.1208731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05_a4d7c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1209109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1209330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1209629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1209769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1209960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwm8l96w
2022-11-23T02:20:42.1210160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwm8l96w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1210441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1210591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1210879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1211018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1211207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iri4knk
2022-11-23T02:20:42.1211405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iri4knk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1211571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1211736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1211901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1212054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1212123Z ok (2.675s)
2022-11-23T02:20:42.1212129Z 
2022-11-23T02:20:42.1212334Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1212412Z Ran 1 test in 2.676s
2022-11-23T02:20:42.1212418Z 
2022-11-23T02:20:42.1212480Z OK
2022-11-23T02:20:42.1212484Z 
2022-11-23T02:20:42.1212571Z Generating XML reports...
2022-11-23T02:20:42.1212926Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015554.xml
2022-11-23T02:20:42.1213206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1213337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1213610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1213750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1213937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2upcd5bh
2022-11-23T02:20:42.1214137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2upcd5bh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1214396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1214402Z 
2022-11-23T02:20:42.1214476Z Running tests...
2022-11-23T02:20:42.1214676Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1215093Z   test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30762
2022-11-23T02:20:42.1215243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30763
2022-11-23T02:20:42.1215401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30764
2022-11-23T02:20:42.1215559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30765
2022-11-23T02:20:42.1215841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1215967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1216314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1216455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1216645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji2fvwar
2022-11-23T02:20:42.1216844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji2fvwar/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1217110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1217273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1217557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1217692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1217879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdlz92pr
2022-11-23T02:20:42.1218077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdlz92pr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1218350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1218474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1218741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1218875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1219063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z5vfjyz
2022-11-23T02:20:42.1219259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z5vfjyz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1219535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1219657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1219939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1220074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1220258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2bjxy9a
2022-11-23T02:20:42.1220441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2bjxy9a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1220610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1220778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1220941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1221107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1221174Z ok (2.778s)
2022-11-23T02:20:42.1221180Z 
2022-11-23T02:20:42.1221378Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1221456Z Ran 1 test in 2.778s
2022-11-23T02:20:42.1221461Z 
2022-11-23T02:20:42.1221511Z OK
2022-11-23T02:20:42.1221529Z 
2022-11-23T02:20:42.1221601Z Generating XML reports...
2022-11-23T02:20:42.1221952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015558.xml
2022-11-23T02:20:42.1222226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1222350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1222633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1222805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1222993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpue_m2onm
2022-11-23T02:20:42.1223189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpue_m2onm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1223433Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1223438Z 
2022-11-23T02:20:42.1223512Z Running tests...
2022-11-23T02:20:42.1223709Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1223968Z   test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30985
2022-11-23T02:20:42.1224127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30986
2022-11-23T02:20:42.1224285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30987
2022-11-23T02:20:42.1224442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30988
2022-11-23T02:20:42.1224723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1224836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1225121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1225261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1225449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02i_l5jd
2022-11-23T02:20:42.1225649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02i_l5jd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1225923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1226049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1226330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1226467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1226639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mu__ktz
2022-11-23T02:20:42.1226834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mu__ktz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1227108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1227234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1227519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1227660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1227847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8hvmcqt
2022-11-23T02:20:42.1228043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8hvmcqt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1228318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1228429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1228710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1228851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1229036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprffw2zzj
2022-11-23T02:20:42.1229232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprffw2zzj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1229429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1229601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1229771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1229920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1229987Z ok (2.578s)
2022-11-23T02:20:42.1229992Z 
2022-11-23T02:20:42.1230192Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1230299Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1230304Z 
2022-11-23T02:20:42.1230366Z OK
2022-11-23T02:20:42.1230371Z 
2022-11-23T02:20:42.1230455Z Generating XML reports...
2022-11-23T02:20:42.1230809Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015603.xml
2022-11-23T02:20:42.1231086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1231211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1231482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1231617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1231802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps92p4i23
2022-11-23T02:20:42.1231995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps92p4i23/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1232255Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1232260Z 
2022-11-23T02:20:42.1232352Z Running tests...
2022-11-23T02:20:42.1232551Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1232780Z   test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31208
2022-11-23T02:20:42.1232942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31209
2022-11-23T02:20:42.1233100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31210
2022-11-23T02:20:42.1233253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31211
2022-11-23T02:20:42.1233530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1233655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1233970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1234111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1234360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphume4je7
2022-11-23T02:20:42.1240248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphume4je7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1240572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1240702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1240983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1241122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1241308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pwr05r_
2022-11-23T02:20:42.1241504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pwr05r_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1241825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1241948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1242216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1242346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1242523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v2jcrjn
2022-11-23T02:20:42.1242712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v2jcrjn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1243013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1243128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1243404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1243532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1243712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12l52yhb
2022-11-23T02:20:42.1243894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12l52yhb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1264689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1264858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1265017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1265169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1265227Z ok (2.577s)
2022-11-23T02:20:42.1265233Z 
2022-11-23T02:20:42.1265432Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1265501Z Ran 1 test in 2.577s
2022-11-23T02:20:42.1265507Z 
2022-11-23T02:20:42.1265554Z OK
2022-11-23T02:20:42.1265564Z 
2022-11-23T02:20:42.1265635Z Generating XML reports...
2022-11-23T02:20:42.1265976Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015608.xml
2022-11-23T02:20:42.1266242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1266359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1266634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1266762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1266939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptljvdyu7
2022-11-23T02:20:42.1267131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptljvdyu7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1267375Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1267386Z 
2022-11-23T02:20:42.1267445Z Running tests...
2022-11-23T02:20:42.1267632Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1267885Z   test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31447
2022-11-23T02:20:42.1268036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31448
2022-11-23T02:20:42.1268184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31449
2022-11-23T02:20:42.1268330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31450
2022-11-23T02:20:42.1268667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1268785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1269053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1269180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1269360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyt7xe8y
2022-11-23T02:20:42.1269547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyt7xe8y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1269884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1270000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1270272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1270402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1270573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpworzgtcb
2022-11-23T02:20:42.1270762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpworzgtcb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1271026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1271141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1271416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1271544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1271723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvec8dspl
2022-11-23T02:20:42.1271912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvec8dspl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1272179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1272288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1272596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1272741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1272916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3x8rix3q
2022-11-23T02:20:42.1273104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3x8rix3q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1273259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1273415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1273571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1273721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1273782Z ok (2.674s)
2022-11-23T02:20:42.1273787Z 
2022-11-23T02:20:42.1273990Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1274058Z Ran 1 test in 2.675s
2022-11-23T02:20:42.1274064Z 
2022-11-23T02:20:42.1274118Z OK
2022-11-23T02:20:42.1274123Z 
2022-11-23T02:20:42.1274198Z Generating XML reports...
2022-11-23T02:20:42.1274545Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015612.xml
2022-11-23T02:20:42.1274813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1274928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1275233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1275365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1275542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7f7ovrb
2022-11-23T02:20:42.1275730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7f7ovrb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1275978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1276027Z 
2022-11-23T02:20:42.1276093Z Running tests...
2022-11-23T02:20:42.1276284Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1276538Z   test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31674
2022-11-23T02:20:42.1276688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31675
2022-11-23T02:20:42.1276831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31676
2022-11-23T02:20:42.1276978Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31677
2022-11-23T02:20:42.1277247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1277362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1277635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1277763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1277940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmaxb8zm
2022-11-23T02:20:42.1278130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmaxb8zm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1278390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1278507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1278782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1278915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1279093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3nsaeb3
2022-11-23T02:20:42.1279281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3nsaeb3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1279545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1279660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1279934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1280054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1280230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_edqjr9
2022-11-23T02:20:42.1280414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_edqjr9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1280677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1280796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1281069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1281194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1281400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_z1j0br
2022-11-23T02:20:42.1281577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_z1j0br/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1281734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1281890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1282044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1282198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1282288Z ok (2.577s)
2022-11-23T02:20:42.1282293Z 
2022-11-23T02:20:42.1282484Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1282553Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1282558Z 
2022-11-23T02:20:42.1282611Z OK
2022-11-23T02:20:42.1282616Z 
2022-11-23T02:20:42.1282687Z Generating XML reports...
2022-11-23T02:20:42.1283026Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015617.xml
2022-11-23T02:20:42.1283291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1283407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1283677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1283802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1283982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78er425m
2022-11-23T02:20:42.1284168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78er425m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1284412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1284423Z 
2022-11-23T02:20:42.1284482Z Running tests...
2022-11-23T02:20:42.1284669Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1284954Z   test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31901
2022-11-23T02:20:42.1285111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31902
2022-11-23T02:20:42.1285260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31903
2022-11-23T02:20:42.1285410Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31904
2022-11-23T02:20:42.1285680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1285796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1286064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1286192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1286369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8tjzv1f
2022-11-23T02:20:42.1286562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8tjzv1f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1286834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1286956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1287238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1287373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1287558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjucq9jc
2022-11-23T02:20:42.1287769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjucq9jc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1288042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1288164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1288444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1288577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1288789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27_2ok8h
2022-11-23T02:20:42.1288980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27_2ok8h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1289250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1289361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1289639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1289773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1289955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgkxi272
2022-11-23T02:20:42.1290147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgkxi272/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1290312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1290476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1290637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1290801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1290855Z ok (2.479s)
2022-11-23T02:20:42.1290860Z 
2022-11-23T02:20:42.1291058Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1291133Z Ran 1 test in 2.479s
2022-11-23T02:20:42.1291138Z 
2022-11-23T02:20:42.1291199Z OK
2022-11-23T02:20:42.1291203Z 
2022-11-23T02:20:42.1291287Z Generating XML reports...
2022-11-23T02:20:42.1291634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015621.xml
2022-11-23T02:20:42.1291906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1292031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1292299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1292434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1292619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8crbi86
2022-11-23T02:20:42.1292813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8crbi86/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1293065Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1293071Z 
2022-11-23T02:20:42.1293142Z Running tests...
2022-11-23T02:20:42.1293338Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1293586Z   test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32128
2022-11-23T02:20:42.1293743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32129
2022-11-23T02:20:42.1293883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32130
2022-11-23T02:20:42.1294065Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32131
2022-11-23T02:20:42.1294342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1294466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1294747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1295047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1295232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0fwrryb
2022-11-23T02:20:42.1295475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0fwrryb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1295750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1295861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1296140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1296273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1296456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g42atr7
2022-11-23T02:20:42.1296648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g42atr7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1296917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1297040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1297321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1297442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1297628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxaa6hxm
2022-11-23T02:20:42.1297823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxaa6hxm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1298093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1298217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1298495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1298628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1298811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4810lf5l
2022-11-23T02:20:42.1299003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4810lf5l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1299155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1299318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1299478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1299639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1299707Z ok (2.678s)
2022-11-23T02:20:42.1299712Z 
2022-11-23T02:20:42.1299909Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1299987Z Ran 1 test in 2.678s
2022-11-23T02:20:42.1299992Z 
2022-11-23T02:20:42.1300053Z OK
2022-11-23T02:20:42.1300057Z 
2022-11-23T02:20:42.1300127Z Generating XML reports...
2022-11-23T02:20:42.1300475Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015625.xml
2022-11-23T02:20:42.1300787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1300911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1301192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1301323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1301508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicxfm6_a
2022-11-23T02:20:42.1301705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicxfm6_a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1302000Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1302006Z 
2022-11-23T02:20:42.1302065Z Running tests...
2022-11-23T02:20:42.1302260Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1302492Z   test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32355
2022-11-23T02:20:42.1302649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32356
2022-11-23T02:20:42.1302806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32357
2022-11-23T02:20:42.1302959Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32358
2022-11-23T02:20:42.1303231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1303354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1303622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1303756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1303942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljixrf1t
2022-11-23T02:20:42.1304138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljixrf1t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1304407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1304529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1304808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1304945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1305129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dn3o1mf
2022-11-23T02:20:42.1305309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dn3o1mf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1305579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1305703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1305984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1306117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1306301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnhomnry
2022-11-23T02:20:42.1306496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnhomnry/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1306768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1306877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1307156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1307317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1307501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrnq1jxs
2022-11-23T02:20:42.1307693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrnq1jxs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1307858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1308022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1308215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1308380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1308434Z ok (2.579s)
2022-11-23T02:20:42.1308439Z 
2022-11-23T02:20:42.1308636Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1308714Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1308720Z 
2022-11-23T02:20:42.1308781Z OK
2022-11-23T02:20:42.1308786Z 
2022-11-23T02:20:42.1308869Z Generating XML reports...
2022-11-23T02:20:42.1309218Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015630.xml
2022-11-23T02:20:42.1309491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1309617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1309885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1310021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1310206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptahbdr9d
2022-11-23T02:20:42.1310405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptahbdr9d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1310665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1310670Z 
2022-11-23T02:20:42.1310743Z Running tests...
2022-11-23T02:20:42.1310940Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1311161Z   test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32578
2022-11-23T02:20:42.1311318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32579
2022-11-23T02:20:42.1311462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32580
2022-11-23T02:20:42.1311616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32581
2022-11-23T02:20:42.1311892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1312017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1312301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1312437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1312623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp557yy99z
2022-11-23T02:20:42.1312817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp557yy99z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1313091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1313204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1313485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1313655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1313840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzmc7h83
2022-11-23T02:20:42.1314036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzmc7h83/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1314308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1314432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1314714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1314865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1315049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2ir9ew3
2022-11-23T02:20:42.1315242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2ir9ew3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1315515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1315637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1315918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1316053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1316237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ytog55x
2022-11-23T02:20:42.1316435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ytog55x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1316587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1316756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1316922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1317084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1317154Z ok (2.575s)
2022-11-23T02:20:42.1317159Z 
2022-11-23T02:20:42.1317356Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1317430Z Ran 1 test in 2.575s
2022-11-23T02:20:42.1317435Z 
2022-11-23T02:20:42.1317495Z OK
2022-11-23T02:20:42.1317500Z 
2022-11-23T02:20:42.1317571Z Generating XML reports...
2022-11-23T02:20:42.1317922Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015634.xml
2022-11-23T02:20:42.1318197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1318323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1318606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1318742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1318929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1fy2vqp
2022-11-23T02:20:42.1319126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1fy2vqp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1319381Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1319386Z 
2022-11-23T02:20:42.1319448Z Running tests...
2022-11-23T02:20:42.1319643Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1319881Z   test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 333
2022-11-23T02:20:42.1320038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 334
2022-11-23T02:20:42.1320221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 335
2022-11-23T02:20:42.1320373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 336
2022-11-23T02:20:42.1320648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1320773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1321042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1321217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1321405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwx0w90f_
2022-11-23T02:20:42.1321600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwx0w90f_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1321875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1321998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1322280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1322417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1322602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpub8925xl
2022-11-23T02:20:42.1322783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpub8925xl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1323058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1323179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1323462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1323596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1323780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfximvon9
2022-11-23T02:20:42.1323975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfximvon9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1324247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1324356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1324641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1324776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1325038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqp6ei8l
2022-11-23T02:20:42.1325237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqp6ei8l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1325401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1325565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1325727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1325891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1325945Z ok (2.575s)
2022-11-23T02:20:42.1325952Z 
2022-11-23T02:20:42.1326153Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1326228Z Ran 1 test in 2.575s
2022-11-23T02:20:42.1326233Z 
2022-11-23T02:20:42.1326294Z OK
2022-11-23T02:20:42.1326299Z 
2022-11-23T02:20:42.1326383Z Generating XML reports...
2022-11-23T02:20:42.1326771Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015639.xml
2022-11-23T02:20:42.1327046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1327169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1327438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1327573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1327762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_otp5_c
2022-11-23T02:20:42.1327989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_otp5_c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1328247Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1328252Z 
2022-11-23T02:20:42.1328324Z Running tests...
2022-11-23T02:20:42.1328524Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1328746Z   test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 556
2022-11-23T02:20:42.1328901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 557
2022-11-23T02:20:42.1329038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 558
2022-11-23T02:20:42.1329191Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 559
2022-11-23T02:20:42.1329468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1329590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1329871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1330013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1330196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywi1rjj3
2022-11-23T02:20:42.1330392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywi1rjj3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1330666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1330775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1331054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1331191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1331377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyed3vhuz
2022-11-23T02:20:42.1331579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyed3vhuz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1331849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1331972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1332255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1332378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1332561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v8_omvr
2022-11-23T02:20:42.1332755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v8_omvr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1333028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1333150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1333461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1333597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1333781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjgh0_6u
2022-11-23T02:20:42.1333974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjgh0_6u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1334124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1334318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1334480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1334644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1334713Z ok (2.580s)
2022-11-23T02:20:42.1334718Z 
2022-11-23T02:20:42.1335029Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1335105Z Ran 1 test in 2.580s
2022-11-23T02:20:42.1335111Z 
2022-11-23T02:20:42.1335172Z OK
2022-11-23T02:20:42.1335177Z 
2022-11-23T02:20:42.1335247Z Generating XML reports...
2022-11-23T02:20:42.1335595Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015643.xml
2022-11-23T02:20:42.1335867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1335996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1336279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1336414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1336602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbqu0_ho
2022-11-23T02:20:42.1336797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbqu0_ho/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1337051Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1337056Z 
2022-11-23T02:20:42.1337115Z Running tests...
2022-11-23T02:20:42.1337314Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1337537Z   test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 781
2022-11-23T02:20:42.1337694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 782
2022-11-23T02:20:42.1337846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 783
2022-11-23T02:20:42.1337998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 784
2022-11-23T02:20:42.1338277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1338401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1338668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1338803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1338988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrr21dgf
2022-11-23T02:20:42.1339185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrr21dgf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1339460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1339582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1339910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1340046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1340231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposwlpbl7
2022-11-23T02:20:42.1340412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposwlpbl7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1340682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1340804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1341119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1341252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1341438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso7nu96m
2022-11-23T02:20:42.1341632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso7nu96m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1341902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1342011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1342290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1342424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1342610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavprxr7r
2022-11-23T02:20:42.1342804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavprxr7r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1342969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1343136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1343297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1343460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1343513Z ok (2.679s)
2022-11-23T02:20:42.1343518Z 
2022-11-23T02:20:42.1343716Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1343792Z Ran 1 test in 2.680s
2022-11-23T02:20:42.1343797Z 
2022-11-23T02:20:42.1343858Z OK
2022-11-23T02:20:42.1343865Z 
2022-11-23T02:20:42.1343949Z Generating XML reports...
2022-11-23T02:20:42.1344296Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015648.xml
2022-11-23T02:20:42.1344569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1344694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1344961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1345096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1345278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71ogldsm
2022-11-23T02:20:42.1345474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71ogldsm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1345730Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1345736Z 
2022-11-23T02:20:42.1345807Z Running tests...
2022-11-23T02:20:42.1346002Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1346235Z   test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1004
2022-11-23T02:20:42.1346418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1005
2022-11-23T02:20:42.1346560Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1006
2022-11-23T02:20:42.1346713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1007
2022-11-23T02:20:42.1346990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1347114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1347399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1347563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1347751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9miybj9n
2022-11-23T02:20:42.1347947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9miybj9n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1348226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1348336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1348616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1348750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1348934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2r98txyu
2022-11-23T02:20:42.1349130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2r98txyu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1349399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1349522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1349804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1349925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1350113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ijv7z2p
2022-11-23T02:20:42.1350310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ijv7z2p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1350580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1350706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1350986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1351122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1351308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn39gb4a
2022-11-23T02:20:42.1351503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn39gb4a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1351653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1351816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1351977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1352137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1352206Z ok (2.678s)
2022-11-23T02:20:42.1352211Z 
2022-11-23T02:20:42.1352407Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1352482Z Ran 1 test in 2.678s
2022-11-23T02:20:42.1352487Z 
2022-11-23T02:20:42.1352547Z OK
2022-11-23T02:20:42.1352552Z 
2022-11-23T02:20:42.1352658Z Generating XML reports...
2022-11-23T02:20:42.1353006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015652.xml
2022-11-23T02:20:42.1353281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1353407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1353686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1353851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1354037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxcteors
2022-11-23T02:20:42.1354234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxcteors/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1354489Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1354494Z 
2022-11-23T02:20:42.1354553Z Running tests...
2022-11-23T02:20:42.1354751Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1354997Z   test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1227
2022-11-23T02:20:42.1355152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1228
2022-11-23T02:20:42.1355305Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1229
2022-11-23T02:20:42.1355458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1230
2022-11-23T02:20:42.1355733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1355854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1356124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1356258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1356445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6cwo7xa
2022-11-23T02:20:42.1356639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6cwo7xa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1356909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1357033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1357313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1357447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1357631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nhgm9f1
2022-11-23T02:20:42.1357811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nhgm9f1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1358082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1358204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1358484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1358620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1358807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzdhupvq
2022-11-23T02:20:42.1359006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzdhupvq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1359310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1359434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1359701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1359834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1360016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee32me15
2022-11-23T02:20:42.1360207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee32me15/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1360401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1360565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1360726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1360889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1360942Z ok (2.678s)
2022-11-23T02:20:42.1360948Z 
2022-11-23T02:20:42.1361146Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1361221Z Ran 1 test in 2.678s
2022-11-23T02:20:42.1361226Z 
2022-11-23T02:20:42.1361287Z OK
2022-11-23T02:20:42.1361292Z 
2022-11-23T02:20:42.1361375Z Generating XML reports...
2022-11-23T02:20:42.1361721Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015657.xml
2022-11-23T02:20:42.1361997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1362120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1362399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1362522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1362706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppytemvvo
2022-11-23T02:20:42.1362899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppytemvvo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1363154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1363159Z 
2022-11-23T02:20:42.1363232Z Running tests...
2022-11-23T02:20:42.1363428Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1363659Z   test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1454
2022-11-23T02:20:42.1363814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1455
2022-11-23T02:20:42.1363953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1456
2022-11-23T02:20:42.1364109Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1457
2022-11-23T02:20:42.1364384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1364509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1364790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1365003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1365195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqujk_j6
2022-11-23T02:20:42.1365391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqujk_j6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1365665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1365807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1366090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1366225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1366412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl_29yl3
2022-11-23T02:20:42.1366607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl_29yl3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1366879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1367043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1367326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1367448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1367634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1pnqhka
2022-11-23T02:20:42.1367829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1pnqhka/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1368101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1368223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1368502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1368638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1368821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdh74_w6d
2022-11-23T02:20:42.1369014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdh74_w6d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1369168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1369372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1369603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1369765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1369832Z ok (2.679s)
2022-11-23T02:20:42.1369837Z 
2022-11-23T02:20:42.1370038Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1370117Z Ran 1 test in 2.679s
2022-11-23T02:20:42.1370123Z 
2022-11-23T02:20:42.1370184Z OK
2022-11-23T02:20:42.1370188Z 
2022-11-23T02:20:42.1370258Z Generating XML reports...
2022-11-23T02:20:42.1370609Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015701.xml
2022-11-23T02:20:42.1370886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1371011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1371289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1371424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1371610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqkyogsj
2022-11-23T02:20:42.1371807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqkyogsj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1372065Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1372070Z 
2022-11-23T02:20:42.1372129Z Running tests...
2022-11-23T02:20:42.1372326Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1372636Z   test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1677
2022-11-23T02:20:42.1372793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1678
2022-11-23T02:20:42.1372947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1679
2022-11-23T02:20:42.1373100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1680
2022-11-23T02:20:42.1373377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1373528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1373798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1373933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1374121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp955_nzfi
2022-11-23T02:20:42.1374316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp955_nzfi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1374590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1374713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1375235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1375374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1375561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmal3lvze
2022-11-23T02:20:42.1375744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmal3lvze/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1376024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1376146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1376426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1376561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1376744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_ifcfyo
2022-11-23T02:20:42.1376938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_ifcfyo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1377211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1377335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1377602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1377738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1377922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5f35e8n
2022-11-23T02:20:42.1378114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5f35e8n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1378277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1378443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1378607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1378770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1378824Z ok (2.578s)
2022-11-23T02:20:42.1378829Z 
2022-11-23T02:20:42.1379028Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1379161Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1379167Z 
2022-11-23T02:20:42.1379226Z OK
2022-11-23T02:20:42.1379231Z 
2022-11-23T02:20:42.1379315Z Generating XML reports...
2022-11-23T02:20:42.1379669Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015706.xml
2022-11-23T02:20:42.1379947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1380072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1380396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1380515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1380701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrvtzju_
2022-11-23T02:20:42.1380902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrvtzju_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1381156Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1381161Z 
2022-11-23T02:20:42.1381235Z Running tests...
2022-11-23T02:20:42.1381432Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1381681Z   test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1900
2022-11-23T02:20:42.1381839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1901
2022-11-23T02:20:42.1381981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1902
2022-11-23T02:20:42.1382138Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1903
2022-11-23T02:20:42.1382416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1382540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1382822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1382961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1383147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89i2z8eb
2022-11-23T02:20:42.1383420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1383546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1383814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1383948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1384139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcjgrrcx
2022-11-23T02:20:42.1384334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89i2z8eb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1384526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcjgrrcx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1384801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1384926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1385206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1385329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1385514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_2cl3mw
2022-11-23T02:20:42.1385733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_2cl3mw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1386007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1386131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1386414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1386548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1386734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qytbqcb
2022-11-23T02:20:42.1386964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qytbqcb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1387117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1387282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1387449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1387609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1387787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.1387962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.1388137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.1388315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.1388617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1388904Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1389199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1389492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1389560Z ok (2.679s)
2022-11-23T02:20:42.1389566Z 
2022-11-23T02:20:42.1389765Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1389841Z Ran 1 test in 2.679s
2022-11-23T02:20:42.1389846Z 
2022-11-23T02:20:42.1389908Z OK
2022-11-23T02:20:42.1389913Z 
2022-11-23T02:20:42.1389997Z Generating XML reports...
2022-11-23T02:20:42.1390349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015710.xml
2022-11-23T02:20:42.1390612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1390739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1391022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1391158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1391341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml5liguq
2022-11-23T02:20:42.1391540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml5liguq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1391795Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1391802Z 
2022-11-23T02:20:42.1391874Z Running tests...
2022-11-23T02:20:42.1392058Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1392329Z   test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2135
2022-11-23T02:20:42.1392489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2136
2022-11-23T02:20:42.1392642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2137
2022-11-23T02:20:42.1392795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2138
2022-11-23T02:20:42.1393067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1393192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1393503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1393638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1393814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnfij941
2022-11-23T02:20:42.1394013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnfij941/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1394286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1394409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1394693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1394830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1395013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc19is2is
2022-11-23T02:20:42.1395209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc19is2is/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1395467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1395592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1395872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1396006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1396189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdxc4mfq
2022-11-23T02:20:42.1396385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdxc4mfq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1396658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1396783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1397063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1397182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1397368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgonblzi
2022-11-23T02:20:42.1397568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgonblzi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1397734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1397894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1398057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1398221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1398289Z ok (2.681s)
2022-11-23T02:20:42.1398294Z 
2022-11-23T02:20:42.1398494Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1398557Z Ran 1 test in 2.682s
2022-11-23T02:20:42.1398562Z 
2022-11-23T02:20:42.1398623Z OK
2022-11-23T02:20:42.1398658Z 
2022-11-23T02:20:42.1398744Z Generating XML reports...
2022-11-23T02:20:42.1399093Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015715.xml
2022-11-23T02:20:42.1399369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1399494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1399779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1399943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1400115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57qdjrtp
2022-11-23T02:20:42.1400312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57qdjrtp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1400566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1400571Z 
2022-11-23T02:20:42.1400643Z Running tests...
2022-11-23T02:20:42.1400839Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1401072Z   test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2358
2022-11-23T02:20:42.1401224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2359
2022-11-23T02:20:42.1401378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2360
2022-11-23T02:20:42.1401533Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2361
2022-11-23T02:20:42.1401796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1401921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1402203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1402340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1402528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsn2nqyb
2022-11-23T02:20:42.1402723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsn2nqyb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1402997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1403121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1403389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1403523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1403711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8q2sr0i
2022-11-23T02:20:42.1403907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8q2sr0i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1404178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1404301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1404581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1404716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1404965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq480jl0i
2022-11-23T02:20:42.1405151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq480jl0i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1405459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1405583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1405862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1405997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1406182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgohqtidx
2022-11-23T02:20:42.1406378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgohqtidx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1406570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1406720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1406885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1407045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1407113Z ok (2.677s)
2022-11-23T02:20:42.1407118Z 
2022-11-23T02:20:42.1407315Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1407390Z Ran 1 test in 2.677s
2022-11-23T02:20:42.1407395Z 
2022-11-23T02:20:42.1407459Z OK
2022-11-23T02:20:42.1407464Z 
2022-11-23T02:20:42.1407548Z Generating XML reports...
2022-11-23T02:20:42.1407897Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015719.xml
2022-11-23T02:20:42.1408160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1408284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1408565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1408702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1408889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8opxk7h
2022-11-23T02:20:42.1409086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8opxk7h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1409341Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1409346Z 
2022-11-23T02:20:42.1409416Z Running tests...
2022-11-23T02:20:42.1409601Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1409845Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2581
2022-11-23T02:20:42.1410001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2582
2022-11-23T02:20:42.1410154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2583
2022-11-23T02:20:42.1410307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2584
2022-11-23T02:20:42.1410581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1410704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1410990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1411127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1411301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzw7g7cyn
2022-11-23T02:20:42.1411498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzw7g7cyn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1411772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1411923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1412208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1412343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1412529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjsossgh1
2022-11-23T02:20:42.1412724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjsossgh1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1412996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1413148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1413427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1413563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1413747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgvgv3k4
2022-11-23T02:20:42.1413940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgvgv3k4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1414210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1414333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1414614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1414739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1415033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9_9tdb8
2022-11-23T02:20:42.1415225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9_9tdb8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1415393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1415557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1415722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1415886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1415953Z ok (2.675s)
2022-11-23T02:20:42.1415958Z 
2022-11-23T02:20:42.1416158Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1416222Z Ran 1 test in 2.675s
2022-11-23T02:20:42.1416227Z 
2022-11-23T02:20:42.1416289Z OK
2022-11-23T02:20:42.1416293Z 
2022-11-23T02:20:42.1416378Z Generating XML reports...
2022-11-23T02:20:42.1416728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015724.xml
2022-11-23T02:20:42.1417005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1417128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1417410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1417546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1417716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfanll7t
2022-11-23T02:20:42.1417912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfanll7t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1418165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1418170Z 
2022-11-23T02:20:42.1418243Z Running tests...
2022-11-23T02:20:42.1418442Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1418760Z   test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2804
2022-11-23T02:20:42.1418917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2805
2022-11-23T02:20:42.1419071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2806
2022-11-23T02:20:42.1419224Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2807
2022-11-23T02:20:42.1419485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1419643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1419924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1420061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1420245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeivz5b88
2022-11-23T02:20:42.1420441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeivz5b88/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1420712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1420834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1421099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1421237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1421420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ud463v2
2022-11-23T02:20:42.1421615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ud463v2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1421889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1422013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1422291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1422428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1422613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfg4tu9c8
2022-11-23T02:20:42.1422794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfg4tu9c8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1423067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1423190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1423468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1423601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1423784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshxdvcun
2022-11-23T02:20:42.1423980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshxdvcun/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1424143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1424295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1424461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1424623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1424698Z 01:57:31 Start training
2022-11-23T02:20:42.1424779Z 01:57:31 PS got 0/3 updates
2022-11-23T02:20:42.1424888Z 01:57:31 PS got 1/3 updates
2022-11-23T02:20:42.1424963Z 01:57:31 PS got 2/3 updates
2022-11-23T02:20:42.1425041Z 01:57:31 PS updated model
2022-11-23T02:20:42.1425103Z 01:57:31 PS got 0/3 updates
2022-11-23T02:20:42.1425177Z 01:57:31 PS got 1/3 updates
2022-11-23T02:20:42.1425248Z 01:57:31 PS got 2/3 updates
2022-11-23T02:20:42.1425324Z 01:57:31 PS updated model
2022-11-23T02:20:42.1425396Z 01:57:31 PS got 0/3 updates
2022-11-23T02:20:42.1425472Z 01:57:31 PS got 1/3 updates
2022-11-23T02:20:42.1425531Z 01:57:31 PS got 2/3 updates
2022-11-23T02:20:42.1425608Z 01:57:31 PS updated model
2022-11-23T02:20:42.1425711Z 01:57:31 PS got 0/3 updates
2022-11-23T02:20:42.1425784Z 01:57:31 PS got 1/3 updates
2022-11-23T02:20:42.1425856Z 01:57:31 PS got 2/3 updates
2022-11-23T02:20:42.1425933Z 01:57:31 PS updated model
2022-11-23T02:20:42.1426010Z 01:57:31 Finish training
2022-11-23T02:20:42.1426099Z 01:57:31 Time spent training: 0.01861191200077883s
2022-11-23T02:20:42.1426195Z 01:57:31 worker2 processing one batch
2022-11-23T02:20:42.1426284Z 01:57:31 worker2 reporting grads
2022-11-23T02:20:42.1426370Z 01:57:31 worker2 got updated model
2022-11-23T02:20:42.1426461Z 01:57:31 worker2 processing one batch
2022-11-23T02:20:42.1426560Z 01:57:31 worker2 reporting grads
2022-11-23T02:20:42.1426645Z 01:57:31 worker2 got updated model
2022-11-23T02:20:42.1426721Z 01:57:31 worker2 processing one batch
2022-11-23T02:20:42.1426807Z 01:57:31 worker2 reporting grads
2022-11-23T02:20:42.1426893Z 01:57:31 worker2 got updated model
2022-11-23T02:20:42.1426981Z 01:57:31 worker2 processing one batch
2022-11-23T02:20:42.1427065Z 01:57:31 worker2 reporting grads
2022-11-23T02:20:42.1427152Z 01:57:31 worker2 got updated model
2022-11-23T02:20:42.1427241Z 01:57:31 worker1 processing one batch
2022-11-23T02:20:42.1427311Z 01:57:31 worker1 reporting grads
2022-11-23T02:20:42.1427402Z 01:57:31 worker1 got updated model
2022-11-23T02:20:42.1427488Z 01:57:31 worker1 processing one batch
2022-11-23T02:20:42.1427572Z 01:57:31 worker1 reporting grads
2022-11-23T02:20:42.1427657Z 01:57:31 worker1 got updated model
2022-11-23T02:20:42.1427748Z 01:57:31 worker1 processing one batch
2022-11-23T02:20:42.1427832Z 01:57:31 worker1 reporting grads
2022-11-23T02:20:42.1427902Z 01:57:31 worker1 got updated model
2022-11-23T02:20:42.1427990Z 01:57:31 worker1 processing one batch
2022-11-23T02:20:42.1428074Z 01:57:31 worker1 reporting grads
2022-11-23T02:20:42.1428158Z 01:57:31 worker1 got updated model
2022-11-23T02:20:42.1428246Z 01:57:31 worker3 processing one batch
2022-11-23T02:20:42.1428329Z 01:57:31 worker3 reporting grads
2022-11-23T02:20:42.1428414Z 01:57:31 worker3 got updated model
2022-11-23T02:20:42.1428492Z 01:57:31 worker3 processing one batch
2022-11-23T02:20:42.1428575Z 01:57:31 worker3 reporting grads
2022-11-23T02:20:42.1428659Z 01:57:31 worker3 got updated model
2022-11-23T02:20:42.1428747Z 01:57:31 worker3 processing one batch
2022-11-23T02:20:42.1428834Z 01:57:31 worker3 reporting grads
2022-11-23T02:20:42.1428919Z 01:57:31 worker3 got updated model
2022-11-23T02:20:42.1429007Z 01:57:31 worker3 processing one batch
2022-11-23T02:20:42.1429077Z 01:57:31 worker3 reporting grads
2022-11-23T02:20:42.1429161Z 01:57:31 worker3 got updated model
2022-11-23T02:20:42.1429228Z ok (2.477s)
2022-11-23T02:20:42.1429234Z 
2022-11-23T02:20:42.1429434Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1429510Z Ran 1 test in 2.478s
2022-11-23T02:20:42.1429515Z 
2022-11-23T02:20:42.1429576Z OK
2022-11-23T02:20:42.1429581Z 
2022-11-23T02:20:42.1429665Z Generating XML reports...
2022-11-23T02:20:42.1430040Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20221123015728.xml
2022-11-23T02:20:42.1430322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1430448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1430764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1430903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1431089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrzquzac
2022-11-23T02:20:42.1431287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrzquzac/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1431550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1431555Z 
2022-11-23T02:20:42.1431659Z Running tests...
2022-11-23T02:20:42.1431846Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1432104Z   test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3027
2022-11-23T02:20:42.1432262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3028
2022-11-23T02:20:42.1432414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3029
2022-11-23T02:20:42.1432565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3030
2022-11-23T02:20:42.1432841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1432967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1433249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1433391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1433566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy0muk6p
2022-11-23T02:20:42.1433765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy0muk6p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1434041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1434164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1434448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1434583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1434768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexulaqk9
2022-11-23T02:20:42.1434963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexulaqk9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1435224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1435347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1435627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1435761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1435946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkhnk4g7
2022-11-23T02:20:42.1436140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkhnk4g7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1436411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1436536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1436823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1436943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1437128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzdw0c01
2022-11-23T02:20:42.1437350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzdw0c01/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1437515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1437678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1437841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1438005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1438143Z Solved! Running reward is now 489.8051609821906!
2022-11-23T02:20:42.1438198Z ok (2.773s)
2022-11-23T02:20:42.1438203Z 
2022-11-23T02:20:42.1438405Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1438481Z Ran 1 test in 2.774s
2022-11-23T02:20:42.1438485Z 
2022-11-23T02:20:42.1438546Z OK
2022-11-23T02:20:42.1438551Z 
2022-11-23T02:20:42.1438636Z Generating XML reports...
2022-11-23T02:20:42.1439051Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20221123015733.xml
2022-11-23T02:20:42.1439326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1439451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1439733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1439858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1440044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm74mk_f_
2022-11-23T02:20:42.1440239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm74mk_f_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1440497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1440502Z 
2022-11-23T02:20:42.1440575Z Running tests...
2022-11-23T02:20:42.1440774Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1441003Z   test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3253
2022-11-23T02:20:42.1441159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3254
2022-11-23T02:20:42.1441422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1441550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1441832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1441968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1442156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw6swqn9
2022-11-23T02:20:42.1442351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw6swqn9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1442624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1442747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1443029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1443152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1443337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcm4yeoa
2022-11-23T02:20:42.1443533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcm4yeoa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1443726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1443893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1443985Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1444419Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-11-23T02:20:42.1444512Z Traceback (most recent call last):
2022-11-23T02:20:42.1444779Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1445006Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1445283Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-11-23T02:20:42.1445431Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-11-23T02:20:42.1445860Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-11-23T02:20:42.1445866Z 
2022-11-23T02:20:42.1445959Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1447444Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2022-11-23T02:20:42.1447541Z Traceback (most recent call last):
2022-11-23T02:20:42.1447807Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1447941Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1448094Z   File "/tmp/tmpm74mk_f_/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-11-23T02:20:42.1448190Z     module = module_rref.local_value()
2022-11-23T02:20:42.1448458Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.1448512Z     raise exc
2022-11-23T02:20:42.1448623Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1449060Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-11-23T02:20:42.1449154Z Traceback (most recent call last):
2022-11-23T02:20:42.1449416Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1449551Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1449824Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-11-23T02:20:42.1449973Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-11-23T02:20:42.1450383Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-11-23T02:20:42.1450404Z 
2022-11-23T02:20:42.1450409Z 
2022-11-23T02:20:42.1450519Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1450947Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-11-23T02:20:42.1451041Z Traceback (most recent call last):
2022-11-23T02:20:42.1451300Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1451432Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1451732Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-11-23T02:20:42.1451878Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-11-23T02:20:42.1452303Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-11-23T02:20:42.1452309Z 
2022-11-23T02:20:42.1452399Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1453873Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2022-11-23T02:20:42.1453970Z Traceback (most recent call last):
2022-11-23T02:20:42.1454238Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1454371Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1454509Z   File "/tmp/tmpm74mk_f_/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-11-23T02:20:42.1454603Z     module = module_rref.local_value()
2022-11-23T02:20:42.1454981Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.1455054Z     raise exc
2022-11-23T02:20:42.1455165Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1455593Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-11-23T02:20:42.1455688Z Traceback (most recent call last):
2022-11-23T02:20:42.1455947Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1456067Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1456341Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 89, in _create_module
2022-11-23T02:20:42.1456487Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-11-23T02:20:42.1456907Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-11-23T02:20:42.1456912Z 
2022-11-23T02:20:42.1456917Z 
2022-11-23T02:20:42.1456985Z ok (2.267s)
2022-11-23T02:20:42.1456990Z 
2022-11-23T02:20:42.1457236Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1457311Z Ran 1 test in 2.268s
2022-11-23T02:20:42.1457316Z 
2022-11-23T02:20:42.1457377Z OK
2022-11-23T02:20:42.1457382Z 
2022-11-23T02:20:42.1457464Z Generating XML reports...
2022-11-23T02:20:42.1457823Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015737.xml
2022-11-23T02:20:42.1458100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1458224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1458558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1458695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1458883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyr3j3dk
2022-11-23T02:20:42.1459085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyr3j3dk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1459344Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1459349Z 
2022-11-23T02:20:42.1459421Z Running tests...
2022-11-23T02:20:42.1459602Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1459839Z   test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3370
2022-11-23T02:20:42.1459997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3371
2022-11-23T02:20:42.1460274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1460398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1460685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1460822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1461009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum6ma602
2022-11-23T02:20:42.1461205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum6ma602/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1461465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1461589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1461872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1462007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1462194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf7su9ee
2022-11-23T02:20:42.1462393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf7su9ee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1462558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1462722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1462984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf7su9ee/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1463254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum6ma602/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1463538Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpum6ma602/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1463607Z ok (2.369s)
2022-11-23T02:20:42.1463642Z 
2022-11-23T02:20:42.1463843Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1463920Z Ran 1 test in 2.370s
2022-11-23T02:20:42.1463925Z 
2022-11-23T02:20:42.1463987Z OK
2022-11-23T02:20:42.1463991Z 
2022-11-23T02:20:42.1464075Z Generating XML reports...
2022-11-23T02:20:42.1464443Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015742.xml
2022-11-23T02:20:42.1464718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1464861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1465144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1465280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1465467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt8bwfjf
2022-11-23T02:20:42.1465663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt8bwfjf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1465918Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1465923Z 
2022-11-23T02:20:42.1465998Z Running tests...
2022-11-23T02:20:42.1466194Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1466420Z   test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3487
2022-11-23T02:20:42.1466580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3488
2022-11-23T02:20:42.1466854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1466981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1467265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1467402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1467589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m57a41t
2022-11-23T02:20:42.1467783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m57a41t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1468057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1468168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1468447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1468582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1468766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdp93tso
2022-11-23T02:20:42.1468962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdp93tso/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1469127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1469291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1469561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m57a41t/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1469833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdp93tso/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1470202Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp9m57a41t/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1470291Z ok (2.366s)
2022-11-23T02:20:42.1470296Z 
2022-11-23T02:20:42.1470499Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1470575Z Ran 1 test in 2.367s
2022-11-23T02:20:42.1470580Z 
2022-11-23T02:20:42.1470641Z OK
2022-11-23T02:20:42.1470646Z 
2022-11-23T02:20:42.1470729Z Generating XML reports...
2022-11-23T02:20:42.1471099Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015746.xml
2022-11-23T02:20:42.1471435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1471560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1471829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1471969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1472156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mrh4fcn
2022-11-23T02:20:42.1472352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mrh4fcn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1472608Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1472613Z 
2022-11-23T02:20:42.1472686Z Running tests...
2022-11-23T02:20:42.1472884Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1473116Z   test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3608
2022-11-23T02:20:42.1473272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3609
2022-11-23T02:20:42.1473540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1473663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1473947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1474085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1474272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj7nys6x
2022-11-23T02:20:42.1474548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1474674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1474954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1475075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1475263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputbhmhpm
2022-11-23T02:20:42.1475460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj7nys6x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1475654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputbhmhpm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1475816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1475978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1476253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj7nys6x/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1476526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputbhmhpm/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1476844Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmputbhmhpm/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1476914Z ok (2.372s)
2022-11-23T02:20:42.1476919Z 
2022-11-23T02:20:42.1477107Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1477182Z Ran 1 test in 2.372s
2022-11-23T02:20:42.1477188Z 
2022-11-23T02:20:42.1477248Z OK
2022-11-23T02:20:42.1477253Z 
2022-11-23T02:20:42.1477335Z Generating XML reports...
2022-11-23T02:20:42.1477704Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015750.xml
2022-11-23T02:20:42.1478009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1478133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1478417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1478540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1478727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxjo6c2i
2022-11-23T02:20:42.1478922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxjo6c2i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1479179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1479184Z 
2022-11-23T02:20:42.1479259Z Running tests...
2022-11-23T02:20:42.1479454Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1479698Z   test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3729
2022-11-23T02:20:42.1479855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3730
2022-11-23T02:20:42.1480134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1480245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1480525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1480662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1480849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8etjffyz
2022-11-23T02:20:42.1481050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8etjffyz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1481320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1481446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1481727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1481862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1482032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqx5r64a
2022-11-23T02:20:42.1482227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqx5r64a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1482389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1482552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1482828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8etjffyz/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1483126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqx5r64a/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1483417Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpyqx5r64a/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1483475Z ok (2.368s)
2022-11-23T02:20:42.1483480Z 
2022-11-23T02:20:42.1483677Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1483751Z Ran 1 test in 2.368s
2022-11-23T02:20:42.1483756Z 
2022-11-23T02:20:42.1483818Z OK
2022-11-23T02:20:42.1483822Z 
2022-11-23T02:20:42.1483934Z Generating XML reports...
2022-11-23T02:20:42.1484300Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015754.xml
2022-11-23T02:20:42.1484576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1484701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1485054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1485177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1485361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xueblwh
2022-11-23T02:20:42.1485556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xueblwh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1485810Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1485818Z 
2022-11-23T02:20:42.1485890Z Running tests...
2022-11-23T02:20:42.1486085Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1486324Z   test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3850
2022-11-23T02:20:42.1486480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3851
2022-11-23T02:20:42.1486739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1486861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1487141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1487276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1487461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny3r_559
2022-11-23T02:20:42.1487653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny3r_559/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1487927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1488051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1488330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1488452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1488634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a154mo0
2022-11-23T02:20:42.1488825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a154mo0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1488987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1489149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1489216Z ok (2.272s)
2022-11-23T02:20:42.1489222Z 
2022-11-23T02:20:42.1489414Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1489487Z Ran 1 test in 2.273s
2022-11-23T02:20:42.1489525Z 
2022-11-23T02:20:42.1489575Z OK
2022-11-23T02:20:42.1489592Z 
2022-11-23T02:20:42.1489662Z Generating XML reports...
2022-11-23T02:20:42.1490027Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015758.xml
2022-11-23T02:20:42.1490301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1490423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1490703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1490868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1491051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf8ht0j5
2022-11-23T02:20:42.1491246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf8ht0j5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1491486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1491504Z 
2022-11-23T02:20:42.1491562Z Running tests...
2022-11-23T02:20:42.1491754Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1491991Z   test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3967
2022-11-23T02:20:42.1492146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3968
2022-11-23T02:20:42.1492422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1492544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1492828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1492963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1493135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp666mw3_n
2022-11-23T02:20:42.1493330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp666mw3_n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1493600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1493722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1494002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1494137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1494321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy30hw5gd
2022-11-23T02:20:42.1494518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy30hw5gd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1494668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1494828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1494983Z ok (2.166s)
2022-11-23T02:20:42.1494989Z 
2022-11-23T02:20:42.1495186Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1495259Z Ran 1 test in 2.166s
2022-11-23T02:20:42.1495264Z 
2022-11-23T02:20:42.1495322Z OK
2022-11-23T02:20:42.1495327Z 
2022-11-23T02:20:42.1495412Z Generating XML reports...
2022-11-23T02:20:42.1495779Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015803.xml
2022-11-23T02:20:42.1496053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1496211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1496496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1496629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1496815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwch0s2p9
2022-11-23T02:20:42.1497010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwch0s2p9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1497266Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1497306Z 
2022-11-23T02:20:42.1497377Z Running tests...
2022-11-23T02:20:42.1497572Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1497816Z   test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4084
2022-11-23T02:20:42.1497972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4085
2022-11-23T02:20:42.1498247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1498368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1498647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1498781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1498969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjvlyp7g
2022-11-23T02:20:42.1499164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjvlyp7g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1499437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1499548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1499828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1499962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1500146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_bpjtnk
2022-11-23T02:20:42.1500338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_bpjtnk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1500500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1500665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1500730Z ok (2.268s)
2022-11-23T02:20:42.1500734Z 
2022-11-23T02:20:42.1500931Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1500991Z Ran 1 test in 2.268s
2022-11-23T02:20:42.1500998Z 
2022-11-23T02:20:42.1501058Z OK
2022-11-23T02:20:42.1501063Z 
2022-11-23T02:20:42.1501145Z Generating XML reports...
2022-11-23T02:20:42.1501514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015807.xml
2022-11-23T02:20:42.1501787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1501909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1502188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1502323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1502495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6pgd79n
2022-11-23T02:20:42.1502754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6pgd79n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1503011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1503016Z 
2022-11-23T02:20:42.1503089Z Running tests...
2022-11-23T02:20:42.1503283Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1503546Z   test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4201
2022-11-23T02:20:42.1503698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4202
2022-11-23T02:20:42.1504013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1504134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1504402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1504536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1504719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp49rsu9y
2022-11-23T02:20:42.1504912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp49rsu9y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1505183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1505304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1505584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1505717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1505888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwmdykvb
2022-11-23T02:20:42.1506085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwmdykvb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1506246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1506408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1506677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwmdykvb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1506948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp49rsu9y/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1507230Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpjwmdykvb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.1507296Z ok (2.267s)
2022-11-23T02:20:42.1507301Z 
2022-11-23T02:20:42.1507498Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1507572Z Ran 1 test in 2.267s
2022-11-23T02:20:42.1507577Z 
2022-11-23T02:20:42.1507623Z OK
2022-11-23T02:20:42.1507628Z 
2022-11-23T02:20:42.1507708Z Generating XML reports...
2022-11-23T02:20:42.1508078Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015811.xml
2022-11-23T02:20:42.1508351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1508476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1508763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1508898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1509110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj1xpyed
2022-11-23T02:20:42.1509308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj1xpyed/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1509549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1509554Z 
2022-11-23T02:20:42.1509624Z Running tests...
2022-11-23T02:20:42.1509819Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1510056Z   test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4318
2022-11-23T02:20:42.1510244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4319
2022-11-23T02:20:42.1510516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1510637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1510917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1511038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1511220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhmltuat
2022-11-23T02:20:42.1511415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhmltuat/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1511690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1511817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1512096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1512230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1512414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t2tn4hs
2022-11-23T02:20:42.1512608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t2tn4hs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1512758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1512919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1512984Z ok (2.270s)
2022-11-23T02:20:42.1512989Z 
2022-11-23T02:20:42.1513185Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1513260Z Ran 1 test in 2.271s
2022-11-23T02:20:42.1513265Z 
2022-11-23T02:20:42.1513323Z OK
2022-11-23T02:20:42.1513327Z 
2022-11-23T02:20:42.1513408Z Generating XML reports...
2022-11-23T02:20:42.1513775Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015815.xml
2022-11-23T02:20:42.1514039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1514162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1514440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1514573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1514756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjdex62b
2022-11-23T02:20:42.1514950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjdex62b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1515206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1515211Z 
2022-11-23T02:20:42.1515282Z Running tests...
2022-11-23T02:20:42.1515474Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1515782Z   test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4435
2022-11-23T02:20:42.1515935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4436
2022-11-23T02:20:42.1516210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1516332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1516612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1516773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1516958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgoay2h6
2022-11-23T02:20:42.1517152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgoay2h6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1517425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1517535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1517812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1517946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1518129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiun44v0
2022-11-23T02:20:42.1518324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiun44v0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1518487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1518648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1518977Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``.
2022-11-23T02:20:42.1519044Z ok (2.271s)
2022-11-23T02:20:42.1519048Z 
2022-11-23T02:20:42.1519232Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1519307Z Ran 1 test in 2.272s
2022-11-23T02:20:42.1519312Z 
2022-11-23T02:20:42.1519371Z OK
2022-11-23T02:20:42.1519375Z 
2022-11-23T02:20:42.1519457Z Generating XML reports...
2022-11-23T02:20:42.1519828Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015819.xml
2022-11-23T02:20:42.1520106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1520227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1520508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1520630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1520816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdhzcugf
2022-11-23T02:20:42.1521012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdhzcugf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1521264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1521271Z 
2022-11-23T02:20:42.1521379Z Running tests...
2022-11-23T02:20:42.1521573Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1521800Z   test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4552
2022-11-23T02:20:42.1521982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4553
2022-11-23T02:20:42.1522257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1522366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1522646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1522781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1522969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3jk0rwz
2022-11-23T02:20:42.1523194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3jk0rwz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1523468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1523590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1523873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1524007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1524178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt_i8h1j
2022-11-23T02:20:42.1524370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt_i8h1j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1524532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1524697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1524763Z ok (2.267s)
2022-11-23T02:20:42.1524767Z 
2022-11-23T02:20:42.1525052Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1525129Z Ran 1 test in 2.268s
2022-11-23T02:20:42.1525134Z 
2022-11-23T02:20:42.1525192Z OK
2022-11-23T02:20:42.1525199Z 
2022-11-23T02:20:42.1525269Z Generating XML reports...
2022-11-23T02:20:42.1525637Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015823.xml
2022-11-23T02:20:42.1525913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1526036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1526318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1526455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1526639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9g185kb
2022-11-23T02:20:42.1526834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9g185kb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1527091Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1527096Z 
2022-11-23T02:20:42.1527155Z Running tests...
2022-11-23T02:20:42.1527349Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1527591Z   test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4669
2022-11-23T02:20:42.1527744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4670
2022-11-23T02:20:42.1528018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1528143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1528423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1528559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1528766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzf_mx_1
2022-11-23T02:20:42.1528963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzf_mx_1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1529235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1529358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1529640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1529805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1529988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac7h13q9
2022-11-23T02:20:42.1530183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac7h13q9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1530347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1530498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1530565Z ok (2.270s)
2022-11-23T02:20:42.1530569Z 
2022-11-23T02:20:42.1530765Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1530841Z Ran 1 test in 2.271s
2022-11-23T02:20:42.1530846Z 
2022-11-23T02:20:42.1530904Z OK
2022-11-23T02:20:42.1530909Z 
2022-11-23T02:20:42.1530991Z Generating XML reports...
2022-11-23T02:20:42.1531362Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015827.xml
2022-11-23T02:20:42.1531636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1531745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1532027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1532161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1532348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf12x09yx
2022-11-23T02:20:42.1532540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf12x09yx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1532794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1532800Z 
2022-11-23T02:20:42.1532871Z Running tests...
2022-11-23T02:20:42.1533066Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1533272Z   test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4786
2022-11-23T02:20:42.1533415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4787
2022-11-23T02:20:42.1533568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4788
2022-11-23T02:20:42.1533718Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4789
2022-11-23T02:20:42.1533993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1534114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1534395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1534532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1534717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsrqhsuq
2022-11-23T02:20:42.1535026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsrqhsuq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1535334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1535460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1535740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1535876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1536060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzrbxyzz
2022-11-23T02:20:42.1536256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzrbxyzz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1536563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1536685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1536952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1537086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1537268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeo7jj02
2022-11-23T02:20:42.1537461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeo7jj02/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1537731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1537853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1538135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1538269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1538454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0p2_tgl
2022-11-23T02:20:42.1538636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0p2_tgl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1538800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1538961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1539122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1539282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1539350Z ok (2.480s)
2022-11-23T02:20:42.1539356Z 
2022-11-23T02:20:42.1539554Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1539627Z Ran 1 test in 2.480s
2022-11-23T02:20:42.1539632Z 
2022-11-23T02:20:42.1539680Z OK
2022-11-23T02:20:42.1539685Z 
2022-11-23T02:20:42.1539771Z Generating XML reports...
2022-11-23T02:20:42.1540112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015832.xml
2022-11-23T02:20:42.1540384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1540506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1540783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1540916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1541101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn31f22gg
2022-11-23T02:20:42.1541297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn31f22gg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1541539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1541544Z 
2022-11-23T02:20:42.1541615Z Running tests...
2022-11-23T02:20:42.1541841Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1542062Z   test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5009
2022-11-23T02:20:42.1542216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5010
2022-11-23T02:20:42.1542367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5011
2022-11-23T02:20:42.1542517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5012
2022-11-23T02:20:42.1542821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1542929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1543213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1543347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1543535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8riqg07
2022-11-23T02:20:42.1543727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8riqg07/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1544001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1544124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1544406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1544542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1544711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsig54xy6
2022-11-23T02:20:42.1544904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsig54xy6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1545175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1545296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1545577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1545711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1545893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaqqtsys
2022-11-23T02:20:42.1546088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaqqtsys/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1546346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1546467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1546746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1546881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1547063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksl0vj4y
2022-11-23T02:20:42.1547255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksl0vj4y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1547417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1547581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1547743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1547890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1547956Z ok (2.474s)
2022-11-23T02:20:42.1547961Z 
2022-11-23T02:20:42.1548187Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1548263Z Ran 1 test in 2.474s
2022-11-23T02:20:42.1548268Z 
2022-11-23T02:20:42.1548327Z OK
2022-11-23T02:20:42.1548332Z 
2022-11-23T02:20:42.1548413Z Generating XML reports...
2022-11-23T02:20:42.1548753Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015836.xml
2022-11-23T02:20:42.1549028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1549176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1549453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1549586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1549768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe_79e19
2022-11-23T02:20:42.1549960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe_79e19/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1550217Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1550222Z 
2022-11-23T02:20:42.1550293Z Running tests...
2022-11-23T02:20:42.1550488Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1550700Z   test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5232
2022-11-23T02:20:42.1550843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5233
2022-11-23T02:20:42.1550993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5234
2022-11-23T02:20:42.1551143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5235
2022-11-23T02:20:42.1551420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1551542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1551823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1551956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1552142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwph9lpw1
2022-11-23T02:20:42.1552323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwph9lpw1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1552597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1552718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1552999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1553131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1553313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbmiesxa
2022-11-23T02:20:42.1553505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbmiesxa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1553775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1553897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1554163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1554295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1554477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hcezdtf
2022-11-23T02:20:42.1554700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hcezdtf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1554971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1555091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1555370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1555504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1555713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gnuzdnv
2022-11-23T02:20:42.1555894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gnuzdnv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1556056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1556221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1556382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1556541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1556606Z ok (2.578s)
2022-11-23T02:20:42.1556611Z 
2022-11-23T02:20:42.1556807Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1556881Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1556886Z 
2022-11-23T02:20:42.1556933Z OK
2022-11-23T02:20:42.1556939Z 
2022-11-23T02:20:42.1557021Z Generating XML reports...
2022-11-23T02:20:42.1557361Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015840.xml
2022-11-23T02:20:42.1557635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1557762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1558037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1558169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1558350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6go81ez9
2022-11-23T02:20:42.1558540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6go81ez9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1558781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1558788Z 
2022-11-23T02:20:42.1558859Z Running tests...
2022-11-23T02:20:42.1559054Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1559269Z   test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5455
2022-11-23T02:20:42.1559425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5456
2022-11-23T02:20:42.1559576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5457
2022-11-23T02:20:42.1559725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5458
2022-11-23T02:20:42.1559996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1560104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1560383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1560520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1560705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck7og87v
2022-11-23T02:20:42.1560926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck7og87v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1561197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1561317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1561596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1561729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1561901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkqf9k8m
2022-11-23T02:20:42.1562128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkqf9k8m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1562397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1562519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1562798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1562931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1563111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qqxtd0b
2022-11-23T02:20:42.1563302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qqxtd0b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1563559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1563681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1563958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1564094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1564277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02jzypmk
2022-11-23T02:20:42.1564469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02jzypmk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1564631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1564794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1565025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1565176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1565244Z ok (2.577s)
2022-11-23T02:20:42.1565249Z 
2022-11-23T02:20:42.1565444Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1565518Z Ran 1 test in 2.577s
2022-11-23T02:20:42.1565523Z 
2022-11-23T02:20:42.1565582Z OK
2022-11-23T02:20:42.1565586Z 
2022-11-23T02:20:42.1565669Z Generating XML reports...
2022-11-23T02:20:42.1566006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015845.xml
2022-11-23T02:20:42.1566279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1566388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1566664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1566797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1566983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52rinmkv
2022-11-23T02:20:42.1567179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52rinmkv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1567465Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1567470Z 
2022-11-23T02:20:42.1567542Z Running tests...
2022-11-23T02:20:42.1567737Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1567957Z   test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5678
2022-11-23T02:20:42.1568100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5679
2022-11-23T02:20:42.1568252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5680
2022-11-23T02:20:42.1568429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5681
2022-11-23T02:20:42.1568702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1568823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1569106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1569238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1569423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_46l62l
2022-11-23T02:20:42.1569600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_46l62l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1569872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1569993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1570273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1570406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1570589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k168czd
2022-11-23T02:20:42.1570782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k168czd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1571052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1571174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1571440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1571571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1571756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo16lnkdx
2022-11-23T02:20:42.1571949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo16lnkdx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1572218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1572341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1572620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1572753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1572939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwk3azdf0
2022-11-23T02:20:42.1573117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwk3azdf0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1573280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1573444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1573606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1573836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1574121Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #5 to worker0, as the agent is shutting down
2022-11-23T02:20:42.1574188Z ok (11.387s)
2022-11-23T02:20:42.1574194Z 
2022-11-23T02:20:42.1574389Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1574452Z Ran 1 test in 11.388s
2022-11-23T02:20:42.1574457Z 
2022-11-23T02:20:42.1574518Z OK
2022-11-23T02:20:42.1574523Z 
2022-11-23T02:20:42.1574603Z Generating XML reports...
2022-11-23T02:20:42.1575050Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015849.xml
2022-11-23T02:20:42.1575377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1575501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1575784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1575919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1576102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ny7jwfb
2022-11-23T02:20:42.1576284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ny7jwfb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1576540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1576546Z 
2022-11-23T02:20:42.1576620Z Running tests...
2022-11-23T02:20:42.1576815Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1577028Z   test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5902
2022-11-23T02:20:42.1577180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5903
2022-11-23T02:20:42.1577334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5904
2022-11-23T02:20:42.1577483Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5905
2022-11-23T02:20:42.1577745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1577867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1578147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1578283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1578468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqhival4
2022-11-23T02:20:42.1578661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqhival4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1578933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1579054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1579336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1579456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1579640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qc9lj08
2022-11-23T02:20:42.1579832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qc9lj08/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1580105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1580226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1580552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1580687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1580870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgaaa6ejr
2022-11-23T02:20:42.1581050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgaaa6ejr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1581318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1581439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1581748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1581880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1582061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpziiuuhme
2022-11-23T02:20:42.1582256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpziiuuhme/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1582419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1582581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1582729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1582891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1582958Z ok (2.572s)
2022-11-23T02:20:42.1582966Z 
2022-11-23T02:20:42.1583165Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1583238Z Ran 1 test in 2.573s
2022-11-23T02:20:42.1583243Z 
2022-11-23T02:20:42.1583302Z OK
2022-11-23T02:20:42.1583307Z 
2022-11-23T02:20:42.1583388Z Generating XML reports...
2022-11-23T02:20:42.1583728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015903.xml
2022-11-23T02:20:42.1583987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1584107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1584385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1584519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1584704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2qr0gnv
2022-11-23T02:20:42.1584899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2qr0gnv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1585155Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1585160Z 
2022-11-23T02:20:42.1585231Z Running tests...
2022-11-23T02:20:42.1585427Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1585633Z   test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6125
2022-11-23T02:20:42.1585787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6126
2022-11-23T02:20:42.1585938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6127
2022-11-23T02:20:42.1586091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6128
2022-11-23T02:20:42.1586364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1586488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1586768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1586930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1587100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k0qyic2
2022-11-23T02:20:42.1587292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k0qyic2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1587563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1587685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1587963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1588126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1588310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74mharjf
2022-11-23T02:20:42.1588505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74mharjf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1588934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1589145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1589639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1589835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1590100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdv54ex5
2022-11-23T02:20:42.1590373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdv54ex5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1590908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1591116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1591403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1591538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1591707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ngocxaf
2022-11-23T02:20:42.1591899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ngocxaf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1592063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1592225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1592387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1592548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1592617Z ok (2.679s)
2022-11-23T02:20:42.1592623Z 
2022-11-23T02:20:42.1592819Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1592880Z Ran 1 test in 2.679s
2022-11-23T02:20:42.1592885Z 
2022-11-23T02:20:42.1592945Z OK
2022-11-23T02:20:42.1592950Z 
2022-11-23T02:20:42.1593031Z Generating XML reports...
2022-11-23T02:20:42.1593372Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015907.xml
2022-11-23T02:20:42.1593642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1593767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1594047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1594181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1594417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b8xhdua
2022-11-23T02:20:42.1594601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b8xhdua/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1594855Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1594859Z 
2022-11-23T02:20:42.1594930Z Running tests...
2022-11-23T02:20:42.1595122Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1595351Z   test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6348
2022-11-23T02:20:42.1595564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6349
2022-11-23T02:20:42.1595716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6350
2022-11-23T02:20:42.1595866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6351
2022-11-23T02:20:42.1596131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1596255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1596542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1596676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1596861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tjgyik1
2022-11-23T02:20:42.1597057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tjgyik1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1597331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1597497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1597781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1597904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1598089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7ucoe6o
2022-11-23T02:20:42.1598284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7ucoe6o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1598554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1598675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1598958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1599091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1599272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98lkc4fa
2022-11-23T02:20:42.1599456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98lkc4fa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1599730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1599851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1600129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1600264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1600448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a93kcdy
2022-11-23T02:20:42.1600641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a93kcdy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1600804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1600997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1601147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1601308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1601374Z ok (2.587s)
2022-11-23T02:20:42.1601379Z 
2022-11-23T02:20:42.1601579Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1601653Z Ran 1 test in 2.587s
2022-11-23T02:20:42.1601658Z 
2022-11-23T02:20:42.1601718Z OK
2022-11-23T02:20:42.1601722Z 
2022-11-23T02:20:42.1601833Z Generating XML reports...
2022-11-23T02:20:42.1602174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015912.xml
2022-11-23T02:20:42.1602434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1602556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1602837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1602975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1603159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xvzvbbq
2022-11-23T02:20:42.1603354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xvzvbbq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1603608Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1603615Z 
2022-11-23T02:20:42.1603686Z Running tests...
2022-11-23T02:20:42.1603881Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1604090Z   test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6571
2022-11-23T02:20:42.1604244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6572
2022-11-23T02:20:42.1604396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6573
2022-11-23T02:20:42.1604546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6574
2022-11-23T02:20:42.1604819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1605007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1605293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1605429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1605601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7hr_1mw
2022-11-23T02:20:42.1605798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7hr_1mw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1606072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1606192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1606471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1606603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1606787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax2cz9ks
2022-11-23T02:20:42.1606983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax2cz9ks/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1607252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1607360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1607669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1607804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1607986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtsvq18f
2022-11-23T02:20:42.1608179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtsvq18f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1608449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1608600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1608882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1609015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1609189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3kkpa15
2022-11-23T02:20:42.1609382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3kkpa15/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1609543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1609708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1609868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1610029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1610114Z ok (2.578s)
2022-11-23T02:20:42.1610120Z 
2022-11-23T02:20:42.1610399Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1610490Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1610497Z 
2022-11-23T02:20:42.1610578Z OK
2022-11-23T02:20:42.1610587Z 
2022-11-23T02:20:42.1610716Z Generating XML reports...
2022-11-23T02:20:42.1611294Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015916.xml
2022-11-23T02:20:42.1611802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1611926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1612206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1612341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1612531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8suw04p
2022-11-23T02:20:42.1612712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8suw04p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1612967Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1612972Z 
2022-11-23T02:20:42.1613045Z Running tests...
2022-11-23T02:20:42.1613243Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1613472Z   test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6794
2022-11-23T02:20:42.1613625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6795
2022-11-23T02:20:42.1613775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6796
2022-11-23T02:20:42.1613927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6797
2022-11-23T02:20:42.1614188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1614310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1614641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1614777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1615067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_27nw05m
2022-11-23T02:20:42.1615259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_27nw05m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1615531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1615653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1615986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1616106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1616287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jpcxitu
2022-11-23T02:20:42.1616483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jpcxitu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1616750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1616871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1617150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1617282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1617468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1ews_wq
2022-11-23T02:20:42.1617737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1617846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1618122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1618255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1618437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdovag__o
2022-11-23T02:20:42.1618627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1ews_wq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1618813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdovag__o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1618979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1619142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1619290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1619453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1619519Z ok (2.577s)
2022-11-23T02:20:42.1619525Z 
2022-11-23T02:20:42.1619718Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1619791Z Ran 1 test in 2.577s
2022-11-23T02:20:42.1619796Z 
2022-11-23T02:20:42.1619855Z OK
2022-11-23T02:20:42.1619860Z 
2022-11-23T02:20:42.1619942Z Generating XML reports...
2022-11-23T02:20:42.1620284Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015921.xml
2022-11-23T02:20:42.1620557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1620669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1620948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1621116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1621303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytv3px6t
2022-11-23T02:20:42.1621497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytv3px6t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1621749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1621755Z 
2022-11-23T02:20:42.1621825Z Running tests...
2022-11-23T02:20:42.1622020Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1622266Z   test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7017
2022-11-23T02:20:42.1622421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7018
2022-11-23T02:20:42.1622572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7019
2022-11-23T02:20:42.1622725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7020
2022-11-23T02:20:42.1622996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1623118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1623397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1623530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1623713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5u8vm5m
2022-11-23T02:20:42.1623896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5u8vm5m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1624167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1624289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1624569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1624701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1624881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d4o15dt
2022-11-23T02:20:42.1625072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d4o15dt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1625343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1625454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1625732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1625865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1626049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp538zo0bw
2022-11-23T02:20:42.1626239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp538zo0bw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1626507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1626627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1626905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1627039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1627208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnau63lh9
2022-11-23T02:20:42.1627398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnau63lh9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1627590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1627816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1627978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1628138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1628204Z ok (2.476s)
2022-11-23T02:20:42.1628209Z 
2022-11-23T02:20:42.1628408Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1628507Z Ran 1 test in 2.476s
2022-11-23T02:20:42.1628525Z 
2022-11-23T02:20:42.1628573Z OK
2022-11-23T02:20:42.1628578Z 
2022-11-23T02:20:42.1628661Z Generating XML reports...
2022-11-23T02:20:42.1628999Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015925.xml
2022-11-23T02:20:42.1629275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1629397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1629676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1629808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1629991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6so01br
2022-11-23T02:20:42.1630171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6so01br/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1630426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1630432Z 
2022-11-23T02:20:42.1630503Z Running tests...
2022-11-23T02:20:42.1630699Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1630928Z   test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7240
2022-11-23T02:20:42.1631082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7241
2022-11-23T02:20:42.1631233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7242
2022-11-23T02:20:42.1631384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7243
2022-11-23T02:20:42.1631648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1631772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1632055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1632194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1632383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejcqh0ic
2022-11-23T02:20:42.1632578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejcqh0ic/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1632851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1632971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1633251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1633372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1633555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o2uynug
2022-11-23T02:20:42.1633750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o2uynug/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1634053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1634176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1634455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1634590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1634774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0ervifu
2022-11-23T02:20:42.1635043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1635178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1635457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1635592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1635778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmptvu591
2022-11-23T02:20:42.1635973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0ervifu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1636162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmptvu591/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1636324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1636487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1636636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1636796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1636863Z ok (2.576s)
2022-11-23T02:20:42.1636868Z 
2022-11-23T02:20:42.1637063Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1637137Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1637144Z 
2022-11-23T02:20:42.1637205Z OK
2022-11-23T02:20:42.1637209Z 
2022-11-23T02:20:42.1637290Z Generating XML reports...
2022-11-23T02:20:42.1637630Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015929.xml
2022-11-23T02:20:42.1637904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1638014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1638291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1638431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1638615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjv4jmnm
2022-11-23T02:20:42.1638812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjv4jmnm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1639066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1639071Z 
2022-11-23T02:20:42.1639142Z Running tests...
2022-11-23T02:20:42.1639338Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1639558Z   test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7463
2022-11-23T02:20:42.1639711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7464
2022-11-23T02:20:42.1639863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7465
2022-11-23T02:20:42.1640016Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7466
2022-11-23T02:20:42.1640288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1640443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1640727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1640880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1641168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8a_o0m6
2022-11-23T02:20:42.1641450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8a_o0m6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1641964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1642276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1642588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1642722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1642908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv7qnnwo
2022-11-23T02:20:42.1643103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv7qnnwo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1643373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1643482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1643763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1643899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1644083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cwmst4_
2022-11-23T02:20:42.1644277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cwmst4_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1644548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1644671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1645082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1645218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1645388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8gvd8v2
2022-11-23T02:20:42.1645585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8gvd8v2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1645747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1645913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1646075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1646237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1646303Z ok (2.577s)
2022-11-23T02:20:42.1646309Z 
2022-11-23T02:20:42.1646507Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1646568Z Ran 1 test in 2.577s
2022-11-23T02:20:42.1646586Z 
2022-11-23T02:20:42.1646633Z OK
2022-11-23T02:20:42.1646638Z 
2022-11-23T02:20:42.1646721Z Generating XML reports...
2022-11-23T02:20:42.1647063Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015934.xml
2022-11-23T02:20:42.1647338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1647460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1647774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1647910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1648094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nwsgg6n
2022-11-23T02:20:42.1648276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nwsgg6n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1648531Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1648536Z 
2022-11-23T02:20:42.1648607Z Running tests...
2022-11-23T02:20:42.1648835Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1649068Z   test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7686
2022-11-23T02:20:42.1649223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7687
2022-11-23T02:20:42.1649377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7688
2022-11-23T02:20:42.1649526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7689
2022-11-23T02:20:42.1649800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1649911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1650191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1650330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1650517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2_gz7h2
2022-11-23T02:20:42.1650712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2_gz7h2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1650987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1651109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1651388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1651508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1651691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7e79m4i0
2022-11-23T02:20:42.1651881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7e79m4i0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1652156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1652277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1652556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1652688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1652870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrog88t1
2022-11-23T02:20:42.1653063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrog88t1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1653321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1653441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1653721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1653853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1654035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniqyqk0w
2022-11-23T02:20:42.1654259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniqyqk0w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1654424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1654585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1654736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1654986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1655054Z ok (2.478s)
2022-11-23T02:20:42.1655107Z 
2022-11-23T02:20:42.1655307Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1655382Z Ran 1 test in 2.479s
2022-11-23T02:20:42.1655387Z 
2022-11-23T02:20:42.1655446Z OK
2022-11-23T02:20:42.1655451Z 
2022-11-23T02:20:42.1655534Z Generating XML reports...
2022-11-23T02:20:42.1655876Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015938.xml
2022-11-23T02:20:42.1656151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1656260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1656542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1656680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1656864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62h5a8x8
2022-11-23T02:20:42.1657059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62h5a8x8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1657311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1657316Z 
2022-11-23T02:20:42.1657386Z Running tests...
2022-11-23T02:20:42.1657583Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1657807Z   test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7909
2022-11-23T02:20:42.1657962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7910
2022-11-23T02:20:42.1658113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7911
2022-11-23T02:20:42.1658263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7912
2022-11-23T02:20:42.1658538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1658659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1658938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1659074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1659259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl18y468a
2022-11-23T02:20:42.1659442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl18y468a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1659712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1659834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1660114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1660248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1660432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv33kkmi
2022-11-23T02:20:42.1660661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv33kkmi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1660934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1661042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1661320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1661452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1661634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yvuy1zo
2022-11-23T02:20:42.1661857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yvuy1zo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1662128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1662249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1662528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1662660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1662828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y1cd7dm
2022-11-23T02:20:42.1663019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y1cd7dm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1663182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1663346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1663507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1663746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1663825Z ok (2.581s)
2022-11-23T02:20:42.1663831Z 
2022-11-23T02:20:42.1664030Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1664092Z Ran 1 test in 2.581s
2022-11-23T02:20:42.1664109Z 
2022-11-23T02:20:42.1664156Z OK
2022-11-23T02:20:42.1664161Z 
2022-11-23T02:20:42.1664243Z Generating XML reports...
2022-11-23T02:20:42.1664580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015943.xml
2022-11-23T02:20:42.1664853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1664977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1665255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1665390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1665576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrfvskp7
2022-11-23T02:20:42.1665760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrfvskp7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1666011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1666016Z 
2022-11-23T02:20:42.1666087Z Running tests...
2022-11-23T02:20:42.1666283Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1666519Z   test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8132
2022-11-23T02:20:42.1666674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8133
2022-11-23T02:20:42.1666824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8134
2022-11-23T02:20:42.1666973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8135
2022-11-23T02:20:42.1667278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1667388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1667668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1667800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1667986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp220surmg
2022-11-23T02:20:42.1668179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp220surmg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1668477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1668599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1668881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1669002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1669185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgmbtffq
2022-11-23T02:20:42.1669379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgmbtffq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1669646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1669767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1670044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1670176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1670358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpagm6eigc
2022-11-23T02:20:42.1670551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpagm6eigc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1670807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1670928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1671208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1671341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1671526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_on1cap
2022-11-23T02:20:42.1671716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_on1cap/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1671878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1672041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1672191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1672350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1672415Z ok (2.480s)
2022-11-23T02:20:42.1672421Z 
2022-11-23T02:20:42.1672620Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1672695Z Ran 1 test in 2.480s
2022-11-23T02:20:42.1672699Z 
2022-11-23T02:20:42.1672759Z OK
2022-11-23T02:20:42.1672766Z 
2022-11-23T02:20:42.1672847Z Generating XML reports...
2022-11-23T02:20:42.1673228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015947.xml
2022-11-23T02:20:42.1673611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1673759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1674043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1674178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1674365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qm_h7us
2022-11-23T02:20:42.1674558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qm_h7us/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1674812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1674880Z 
2022-11-23T02:20:42.1674952Z Running tests...
2022-11-23T02:20:42.1675149Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1675368Z   test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8355
2022-11-23T02:20:42.1675522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8356
2022-11-23T02:20:42.1675674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8357
2022-11-23T02:20:42.1675822Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8358
2022-11-23T02:20:42.1676096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1676219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1676504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1676637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1676821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4q_8srb
2022-11-23T02:20:42.1677004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4q_8srb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1677274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1677395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1677674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1677808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1677990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsnr5kk7
2022-11-23T02:20:42.1678187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsnr5kk7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1678459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1678570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1678846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1678980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1679162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ug4ux5b
2022-11-23T02:20:42.1679354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ug4ux5b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1679621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1679745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1680022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1680156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1680355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgjb2o3h
2022-11-23T02:20:42.1680551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgjb2o3h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1680715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1680878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1681037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1681223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1681289Z ok (2.578s)
2022-11-23T02:20:42.1681294Z 
2022-11-23T02:20:42.1681491Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1681552Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1681570Z 
2022-11-23T02:20:42.1681617Z OK
2022-11-23T02:20:42.1681621Z 
2022-11-23T02:20:42.1681705Z Generating XML reports...
2022-11-23T02:20:42.1682046Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015951.xml
2022-11-23T02:20:42.1682319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1682440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1682721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1682857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1683039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqgme_vi
2022-11-23T02:20:42.1683221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqgme_vi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1683475Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1683481Z 
2022-11-23T02:20:42.1683551Z Running tests...
2022-11-23T02:20:42.1683745Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1683978Z   test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8578
2022-11-23T02:20:42.1684133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8579
2022-11-23T02:20:42.1684283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8580
2022-11-23T02:20:42.1684434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8581
2022-11-23T02:20:42.1684708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1684817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1685190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1685325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1685509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7v5fmrv
2022-11-23T02:20:42.1685703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7v5fmrv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1685976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1686099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1686383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1686506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1686732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts2ze4e3
2022-11-23T02:20:42.1686926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts2ze4e3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1687199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1687319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1687600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1687732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1687962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_v_8wsf
2022-11-23T02:20:42.1688151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_v_8wsf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1688409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1688530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1688806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1688939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1689120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3jhz1dh
2022-11-23T02:20:42.1689311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3jhz1dh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1689477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1689637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1689785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1689948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1690013Z ok (2.476s)
2022-11-23T02:20:42.1690018Z 
2022-11-23T02:20:42.1690215Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1690290Z Ran 1 test in 2.476s
2022-11-23T02:20:42.1690295Z 
2022-11-23T02:20:42.1690354Z OK
2022-11-23T02:20:42.1690359Z 
2022-11-23T02:20:42.1690442Z Generating XML reports...
2022-11-23T02:20:42.1690781Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015956.xml
2022-11-23T02:20:42.1691056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1691166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1691443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1691579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1691763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94tkl157
2022-11-23T02:20:42.1691957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94tkl157/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1692211Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1692217Z 
2022-11-23T02:20:42.1692287Z Running tests...
2022-11-23T02:20:42.1692480Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1692705Z   test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8801
2022-11-23T02:20:42.1692858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8802
2022-11-23T02:20:42.1693008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8803
2022-11-23T02:20:42.1693188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8804
2022-11-23T02:20:42.1693462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1693583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1693861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1693994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1694210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7hjvsnz
2022-11-23T02:20:42.1694394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7hjvsnz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1694663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1694785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1695184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1695320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1695502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tm95udn
2022-11-23T02:20:42.1695693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tm95udn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1695962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1696087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1696351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1696485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1696667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk0vfg36
2022-11-23T02:20:42.1696860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk0vfg36/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1697129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1697250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1697528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1697662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1697831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxnvla8l
2022-11-23T02:20:42.1698023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxnvla8l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1698187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1698350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1698510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1698669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1698735Z ok (2.581s)
2022-11-23T02:20:42.1698740Z 
2022-11-23T02:20:42.1698938Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1699002Z Ran 1 test in 2.581s
2022-11-23T02:20:42.1699019Z 
2022-11-23T02:20:42.1699066Z OK
2022-11-23T02:20:42.1699070Z 
2022-11-23T02:20:42.1699152Z Generating XML reports...
2022-11-23T02:20:42.1699490Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020000.xml
2022-11-23T02:20:42.1699812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1699936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1700212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1700346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1700529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj3u_c5a
2022-11-23T02:20:42.1700744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj3u_c5a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1700998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1701003Z 
2022-11-23T02:20:42.1701074Z Running tests...
2022-11-23T02:20:42.1701270Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1701496Z   test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9024
2022-11-23T02:20:42.1701650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9025
2022-11-23T02:20:42.1701840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9026
2022-11-23T02:20:42.1702017Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9027
2022-11-23T02:20:42.1702294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1702405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1702686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1702819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1703004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhsen7rm
2022-11-23T02:20:42.1703198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhsen7rm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1703471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1703592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1703870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1703992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1704176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgvww36x
2022-11-23T02:20:42.1704371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgvww36x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1704643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1704770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1705046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1705178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1705360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms1949h4
2022-11-23T02:20:42.1705551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms1949h4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1705812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1705932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1706242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1706375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1706555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ujjx8fg
2022-11-23T02:20:42.1706746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ujjx8fg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1706908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1707071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1707247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1707408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1707473Z ok (2.579s)
2022-11-23T02:20:42.1707478Z 
2022-11-23T02:20:42.1707675Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1707750Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1707755Z 
2022-11-23T02:20:42.1707815Z OK
2022-11-23T02:20:42.1707820Z 
2022-11-23T02:20:42.1707901Z Generating XML reports...
2022-11-23T02:20:42.1708239Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020005.xml
2022-11-23T02:20:42.1708509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1708619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1708897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1709030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1709213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfym38gjk
2022-11-23T02:20:42.1709408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfym38gjk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1709663Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1709668Z 
2022-11-23T02:20:42.1709738Z Running tests...
2022-11-23T02:20:42.1709933Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1710151Z   test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9247
2022-11-23T02:20:42.1710308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9248
2022-11-23T02:20:42.1710461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9249
2022-11-23T02:20:42.1710611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9250
2022-11-23T02:20:42.1710881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1711005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1711282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1711416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1711600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxutuknx0
2022-11-23T02:20:42.1711782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxutuknx0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1712053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1712174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1712453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1712618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1712802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbla1pvo
2022-11-23T02:20:42.1712997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbla1pvo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1713267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1713387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1713652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1713813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1713994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wj4d7qu
2022-11-23T02:20:42.1714188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wj4d7qu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1714491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1714668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1715111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1715361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1715690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv95t1sv
2022-11-23T02:20:42.1715928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv95t1sv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1716091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1716253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1716419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1716582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1716649Z ok (2.589s)
2022-11-23T02:20:42.1716655Z 
2022-11-23T02:20:42.1716854Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1716916Z Ran 1 test in 2.589s
2022-11-23T02:20:42.1716933Z 
2022-11-23T02:20:42.1716980Z OK
2022-11-23T02:20:42.1716985Z 
2022-11-23T02:20:42.1717066Z Generating XML reports...
2022-11-23T02:20:42.1717408Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020009.xml
2022-11-23T02:20:42.1717679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1717802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1718081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1718212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1726612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0twt16k1
2022-11-23T02:20:42.1726883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0twt16k1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1727201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1727217Z 
2022-11-23T02:20:42.1727278Z Running tests...
2022-11-23T02:20:42.1727476Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1727698Z   test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9470
2022-11-23T02:20:42.1727854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9471
2022-11-23T02:20:42.1728099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9472
2022-11-23T02:20:42.1728253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9473
2022-11-23T02:20:42.1728551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1728727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1729009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1729183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1729359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx_6g8q0
2022-11-23T02:20:42.1729623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1729739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1730011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1730137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1730314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_q9wqxy
2022-11-23T02:20:42.1730500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx_6g8q0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1730676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_q9wqxy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1730943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1731058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1731331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1731456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1731632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2yqnysa
2022-11-23T02:20:42.1731816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2yqnysa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1732082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1732205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1732475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1732607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1732790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgcbfw6d
2022-11-23T02:20:42.1732983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgcbfw6d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1733146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1733308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1733466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1733626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1733707Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1733828Z RuntimeError('Expected error')
2022-11-23T02:20:42.1733915Z Traceback (most recent call last):
2022-11-23T02:20:42.1734176Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1734308Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1734592Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1734673Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1735070Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1735159Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1735246Z RuntimeError: Expected error
2022-11-23T02:20:42.1735252Z 
2022-11-23T02:20:42.1735342Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1735464Z RuntimeError('Expected error')
2022-11-23T02:20:42.1735607Z Traceback (most recent call last):
2022-11-23T02:20:42.1735865Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1735996Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1736248Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1736317Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1736606Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1736705Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1736793Z RuntimeError: Expected error
2022-11-23T02:20:42.1736798Z 
2022-11-23T02:20:42.1736887Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.1737009Z RuntimeError('Expected error')
2022-11-23T02:20:42.1737099Z Traceback (most recent call last):
2022-11-23T02:20:42.1737354Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1737473Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1737722Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1737805Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1738092Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1738189Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1738275Z RuntimeError: Expected error
2022-11-23T02:20:42.1738280Z 
2022-11-23T02:20:42.1738366Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.1738475Z RuntimeError('Expected error')
2022-11-23T02:20:42.1738562Z Traceback (most recent call last):
2022-11-23T02:20:42.1738814Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1738945Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1739193Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1739273Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1739555Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1739652Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1739726Z RuntimeError: Expected error
2022-11-23T02:20:42.1739743Z 
2022-11-23T02:20:42.1739796Z ok (2.582s)
2022-11-23T02:20:42.1739801Z 
2022-11-23T02:20:42.1739998Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1740070Z Ran 1 test in 2.583s
2022-11-23T02:20:42.1740075Z 
2022-11-23T02:20:42.1740135Z OK
2022-11-23T02:20:42.1740139Z 
2022-11-23T02:20:42.1740220Z Generating XML reports...
2022-11-23T02:20:42.1740562Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020014.xml
2022-11-23T02:20:42.1740831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1740955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1741261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1741399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1741578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h2gycwv
2022-11-23T02:20:42.1741771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h2gycwv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1742024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1742077Z 
2022-11-23T02:20:42.1742147Z Running tests...
2022-11-23T02:20:42.1742341Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1743035Z   test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.255s)
2022-11-23T02:20:42.1743041Z 
2022-11-23T02:20:42.1743237Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1743310Z Ran 1 test in 0.255s
2022-11-23T02:20:42.1743315Z 
2022-11-23T02:20:42.1743374Z OK (skipped=1)
2022-11-23T02:20:42.1743378Z 
2022-11-23T02:20:42.1743462Z Generating XML reports...
2022-11-23T02:20:42.1743800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020018.xml
2022-11-23T02:20:42.1744076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1744197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1744514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1744690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1744894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7i_1kkb
2022-11-23T02:20:42.1745077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7i_1kkb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1745334Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1745339Z 
2022-11-23T02:20:42.1745408Z Running tests...
2022-11-23T02:20:42.1745606Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1745833Z   test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9703
2022-11-23T02:20:42.1745986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9704
2022-11-23T02:20:42.1746137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9705
2022-11-23T02:20:42.1746286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9706
2022-11-23T02:20:42.1746560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1746670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1746951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1747087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1747273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx3592jr
2022-11-23T02:20:42.1747465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx3592jr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1747774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1747899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1748177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1748309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1748479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms_gu4yv
2022-11-23T02:20:42.1748672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms_gu4yv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1748969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1749088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1749363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1749497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1749680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz44kg_5h
2022-11-23T02:20:42.1749871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz44kg_5h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1750129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1750248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1750526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1750662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1750843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4qdrcye
2022-11-23T02:20:42.1751037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4qdrcye/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1751200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1751363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1751523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1751671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1751763Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.1751886Z RuntimeError('Expected error')
2022-11-23T02:20:42.1751975Z Traceback (most recent call last):
2022-11-23T02:20:42.1752236Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1752371Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1752624Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1752692Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1752980Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1753078Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1753164Z RuntimeError: Expected error
2022-11-23T02:20:42.1753169Z 
2022-11-23T02:20:42.1753257Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.1753376Z RuntimeError('Expected error')
2022-11-23T02:20:42.1753465Z Traceback (most recent call last):
2022-11-23T02:20:42.1753721Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1753839Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1754087Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1754195Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1754482Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1754580Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1754665Z RuntimeError: Expected error
2022-11-23T02:20:42.1754670Z 
2022-11-23T02:20:42.1754760Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1754919Z RuntimeError('Expected error')
2022-11-23T02:20:42.1755015Z Traceback (most recent call last):
2022-11-23T02:20:42.1755271Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1755433Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1755681Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1755762Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1756049Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1756149Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1756235Z RuntimeError: Expected error
2022-11-23T02:20:42.1756240Z 
2022-11-23T02:20:42.1756316Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1756438Z RuntimeError('Expected error')
2022-11-23T02:20:42.1756526Z Traceback (most recent call last):
2022-11-23T02:20:42.1756778Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1756912Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1757156Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 163, in wrapper
2022-11-23T02:20:42.1757238Z     return fn(*args, **kwargs)
2022-11-23T02:20:42.1757523Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 481, in async_raise_func
2022-11-23T02:20:42.1757611Z     raise RuntimeError("Expected error")
2022-11-23T02:20:42.1757696Z RuntimeError: Expected error
2022-11-23T02:20:42.1757701Z 
2022-11-23T02:20:42.1757768Z ok (2.580s)
2022-11-23T02:20:42.1757773Z 
2022-11-23T02:20:42.1757971Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1758042Z Ran 1 test in 2.580s
2022-11-23T02:20:42.1758047Z 
2022-11-23T02:20:42.1758106Z OK
2022-11-23T02:20:42.1758111Z 
2022-11-23T02:20:42.1758191Z Generating XML reports...
2022-11-23T02:20:42.1758526Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020020.xml
2022-11-23T02:20:42.1758790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1758913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1759196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1759328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1759513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_irkkya
2022-11-23T02:20:42.1759706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_irkkya/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1759958Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1759963Z 
2022-11-23T02:20:42.1760033Z Running tests...
2022-11-23T02:20:42.1760230Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1760440Z   test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9926
2022-11-23T02:20:42.1760593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9927
2022-11-23T02:20:42.1760772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9928
2022-11-23T02:20:42.1760922Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9929
2022-11-23T02:20:42.1761196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1761317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1761598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1761730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1761931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhfys5n7
2022-11-23T02:20:42.1762126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhfys5n7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1762399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1762520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1762801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1762933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1763118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0s4iypd
2022-11-23T02:20:42.1763311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0s4iypd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1763582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1763692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1763969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1764105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1764341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq6m1lwh
2022-11-23T02:20:42.1764562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq6m1lwh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1764834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1765025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1765311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1765431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1765614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60ussqbn
2022-11-23T02:20:42.1765810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60ussqbn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1765974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1766138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1766300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1766460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1766528Z ok (2.479s)
2022-11-23T02:20:42.1766535Z 
2022-11-23T02:20:42.1766733Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1766795Z Ran 1 test in 2.479s
2022-11-23T02:20:42.1766800Z 
2022-11-23T02:20:42.1766859Z OK
2022-11-23T02:20:42.1766863Z 
2022-11-23T02:20:42.1766945Z Generating XML reports...
2022-11-23T02:20:42.1767326Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020025.xml
2022-11-23T02:20:42.1767601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1767723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1767999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1768134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1768305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfffo5do7
2022-11-23T02:20:42.1768529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfffo5do7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1768782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1768787Z 
2022-11-23T02:20:42.1768857Z Running tests...
2022-11-23T02:20:42.1769056Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1769290Z   test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10149
2022-11-23T02:20:42.1769444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10150
2022-11-23T02:20:42.1769595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10151
2022-11-23T02:20:42.1769745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10152
2022-11-23T02:20:42.1770012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1770132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1770410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1770544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1770728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxggpwkkp
2022-11-23T02:20:42.1770925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxggpwkkp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1771195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1771314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1771592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1771714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1771896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntyd6f3d
2022-11-23T02:20:42.1772088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntyd6f3d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1772358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1772478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1772755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1772886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1773067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n7o7q95
2022-11-23T02:20:42.1773245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n7o7q95/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1773515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1773634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1773937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1774068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1774252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc4uz38l
2022-11-23T02:20:42.1774444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc4uz38l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1774606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1774796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1775060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1775222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1775288Z ok (2.579s)
2022-11-23T02:20:42.1775293Z 
2022-11-23T02:20:42.1775493Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1775567Z Ran 1 test in 2.579s
2022-11-23T02:20:42.1775572Z 
2022-11-23T02:20:42.1775635Z OK
2022-11-23T02:20:42.1775639Z 
2022-11-23T02:20:42.1775723Z Generating XML reports...
2022-11-23T02:20:42.1776065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020029.xml
2022-11-23T02:20:42.1776326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1776451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1776730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1776864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1777050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf47320b_
2022-11-23T02:20:42.1777239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf47320b_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1777494Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1777500Z 
2022-11-23T02:20:42.1777571Z Running tests...
2022-11-23T02:20:42.1777767Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1777997Z   test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10372
2022-11-23T02:20:42.1778153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10373
2022-11-23T02:20:42.1778307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10374
2022-11-23T02:20:42.1778459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10375
2022-11-23T02:20:42.1778735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1778855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1779135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1779271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1779442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplahocra5
2022-11-23T02:20:42.1779638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplahocra5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1779910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1780033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1780363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1780497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1780682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0pu27bz
2022-11-23T02:20:42.1780878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0pu27bz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1781148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1781256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1781572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1781707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1781889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso9mtppy
2022-11-23T02:20:42.1782087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso9mtppy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1782358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1782479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1782759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1782890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1783062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41339ccy
2022-11-23T02:20:42.1783255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41339ccy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1783418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1783581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1783741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1783900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1783965Z ok (2.488s)
2022-11-23T02:20:42.1783970Z 
2022-11-23T02:20:42.1784163Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1784224Z Ran 1 test in 2.488s
2022-11-23T02:20:42.1784229Z 
2022-11-23T02:20:42.1784290Z OK
2022-11-23T02:20:42.1784297Z 
2022-11-23T02:20:42.1784379Z Generating XML reports...
2022-11-23T02:20:42.1784718Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020034.xml
2022-11-23T02:20:42.1784987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1785109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1785385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1785520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1785692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqstcjq4
2022-11-23T02:20:42.1785888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqstcjq4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1786138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1786145Z 
2022-11-23T02:20:42.1786215Z Running tests...
2022-11-23T02:20:42.1786409Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1786712Z   test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10595
2022-11-23T02:20:42.1786869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10596
2022-11-23T02:20:42.1787024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10597
2022-11-23T02:20:42.1787175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10598
2022-11-23T02:20:42.1787438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1787566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1787887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1788021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1788205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2c_q2pa
2022-11-23T02:20:42.1788398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2c_q2pa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1788671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1788792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1789070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1789191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1789372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkig9yk3l
2022-11-23T02:20:42.1789566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkig9yk3l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1789832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1789955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1790231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1790362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1790546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoggcrbgp
2022-11-23T02:20:42.1790726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoggcrbgp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1791003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1791127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1791409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1791542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1791726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw6cr2m1
2022-11-23T02:20:42.1791919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw6cr2m1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1792081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1792242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1792389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1792553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1792617Z ok (2.579s)
2022-11-23T02:20:42.1792622Z 
2022-11-23T02:20:42.1792814Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1792885Z Ran 1 test in 2.580s
2022-11-23T02:20:42.1792890Z 
2022-11-23T02:20:42.1792950Z OK
2022-11-23T02:20:42.1792954Z 
2022-11-23T02:20:42.1793063Z Generating XML reports...
2022-11-23T02:20:42.1793398Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020038.xml
2022-11-23T02:20:42.1793657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1793777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1794057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1794220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1794403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt45pf6on
2022-11-23T02:20:42.1794595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt45pf6on/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1794851Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1794856Z 
2022-11-23T02:20:42.1794924Z Running tests...
2022-11-23T02:20:42.1795117Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1795346Z   test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10818
2022-11-23T02:20:42.1795500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10819
2022-11-23T02:20:42.1795653Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10820
2022-11-23T02:20:42.1795806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10821
2022-11-23T02:20:42.1796079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1796199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1796484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1796616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1796788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejzg5biv
2022-11-23T02:20:42.1796984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejzg5biv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1797254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1797375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1797656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1797789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1797975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvybqfty
2022-11-23T02:20:42.1798170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvybqfty/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1798438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1798548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1798825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1798957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1799141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevivzjqd
2022-11-23T02:20:42.1799336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevivzjqd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1799643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1799765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1800046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1800179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1800349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3woz4reo
2022-11-23T02:20:42.1800543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3woz4reo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1800737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1800898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1801059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1801218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1801283Z ok (2.581s)
2022-11-23T02:20:42.1801288Z 
2022-11-23T02:20:42.1801483Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1801543Z Ran 1 test in 2.581s
2022-11-23T02:20:42.1801548Z 
2022-11-23T02:20:42.1801605Z OK
2022-11-23T02:20:42.1801610Z 
2022-11-23T02:20:42.1801690Z Generating XML reports...
2022-11-23T02:20:42.1802029Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020043.xml
2022-11-23T02:20:42.1802302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1802424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1802704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1802840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1803023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqlaee_o
2022-11-23T02:20:42.1803208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqlaee_o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1803461Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1803466Z 
2022-11-23T02:20:42.1803535Z Running tests...
2022-11-23T02:20:42.1803725Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1803967Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11041
2022-11-23T02:20:42.1804121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11042
2022-11-23T02:20:42.1804275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11043
2022-11-23T02:20:42.1804425Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11044
2022-11-23T02:20:42.1804687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1804807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1805175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1805310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1805495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27lrx0hw
2022-11-23T02:20:42.1805687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27lrx0hw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1805958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1806114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1806394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1806514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1806698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq87h4tdp
2022-11-23T02:20:42.1806890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq87h4tdp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1807191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1807312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1807590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1807724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1807904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps51l5grc
2022-11-23T02:20:42.1808084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps51l5grc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1808355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1808478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1808755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1808892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1809073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfjfzogw
2022-11-23T02:20:42.1809267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfjfzogw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1809427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1809591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1809739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1809899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1809963Z ok (2.482s)
2022-11-23T02:20:42.1809968Z 
2022-11-23T02:20:42.1810162Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1810237Z Ran 1 test in 2.482s
2022-11-23T02:20:42.1810242Z 
2022-11-23T02:20:42.1810299Z OK
2022-11-23T02:20:42.1810304Z 
2022-11-23T02:20:42.1810384Z Generating XML reports...
2022-11-23T02:20:42.1810721Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020047.xml
2022-11-23T02:20:42.1810983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1811104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1811382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1811512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1811693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84yryrvj
2022-11-23T02:20:42.1811885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84yryrvj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1812136Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1812141Z 
2022-11-23T02:20:42.1812211Z Running tests...
2022-11-23T02:20:42.1812432Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1812656Z   test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11264
2022-11-23T02:20:42.1812811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11265
2022-11-23T02:20:42.1812962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11266
2022-11-23T02:20:42.1813113Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11267
2022-11-23T02:20:42.1813382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1813531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1813809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1813941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1814124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcka8k4w
2022-11-23T02:20:42.1814306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcka8k4w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1814575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1814698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1815081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1815218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1815400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89k59m5x
2022-11-23T02:20:42.1815589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89k59m5x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1815860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1815969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1816246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1816380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1816563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvpgiqp0
2022-11-23T02:20:42.1816756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvpgiqp0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1817030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1817153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1817432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1817563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1817731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzhl1uqf
2022-11-23T02:20:42.1817922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzhl1uqf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1818081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1818243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1818405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1818565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1818631Z ok (2.584s)
2022-11-23T02:20:42.1818638Z 
2022-11-23T02:20:42.1818880Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1818943Z Ran 1 test in 2.585s
2022-11-23T02:20:42.1818948Z 
2022-11-23T02:20:42.1819007Z OK
2022-11-23T02:20:42.1819011Z 
2022-11-23T02:20:42.1819094Z Generating XML reports...
2022-11-23T02:20:42.1819428Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020051.xml
2022-11-23T02:20:42.1819701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1819822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1820140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1820273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1820454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37rgyu7z
2022-11-23T02:20:42.1820639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37rgyu7z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1820897Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1820902Z 
2022-11-23T02:20:42.1820973Z Running tests...
2022-11-23T02:20:42.1821169Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1821415Z   test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11487
2022-11-23T02:20:42.1821572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11488
2022-11-23T02:20:42.1821724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11489
2022-11-23T02:20:42.1821876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11490
2022-11-23T02:20:42.1822138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1822258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1822537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1822669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1822855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsykwyzt
2022-11-23T02:20:42.1823047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsykwyzt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1823316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1823437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1823715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1823836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1824015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcozio9mq
2022-11-23T02:20:42.1824207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcozio9mq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1824475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1824595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1824874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1825004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1825185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_nmn7nc
2022-11-23T02:20:42.1825394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_nmn7nc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1825662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1825783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1826057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1826188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1826371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml9rpv7a
2022-11-23T02:20:42.1826598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml9rpv7a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1826758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1826922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1827070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1827229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1827296Z ok (3.477s)
2022-11-23T02:20:42.1827301Z 
2022-11-23T02:20:42.1827495Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1827566Z Ran 1 test in 3.478s
2022-11-23T02:20:42.1827571Z 
2022-11-23T02:20:42.1827628Z OK
2022-11-23T02:20:42.1827633Z 
2022-11-23T02:20:42.1827718Z Generating XML reports...
2022-11-23T02:20:42.1828057Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020056.xml
2022-11-23T02:20:42.1828327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1828436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1828715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1828846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1829026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhzj8lem
2022-11-23T02:20:42.1829219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhzj8lem/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1829470Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1829477Z 
2022-11-23T02:20:42.1829547Z Running tests...
2022-11-23T02:20:42.1829739Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1829962Z   test_async_record_function_legacy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11710
2022-11-23T02:20:42.1830120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11711
2022-11-23T02:20:42.1830272Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11712
2022-11-23T02:20:42.1830422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11713
2022-11-23T02:20:42.1830691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1830811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1831091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1831226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1831412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzqv3ng_
2022-11-23T02:20:42.1831621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzqv3ng_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1831891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1832013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1832287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1832418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1832595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mfk7i2j
2022-11-23T02:20:42.1832829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mfk7i2j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1833096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1833204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1833482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1833613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1833792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthb9bmcq
2022-11-23T02:20:42.1833983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthb9bmcq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1834252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1834373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1834648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1834779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1834949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptoev3_fd
2022-11-23T02:20:42.1835138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptoev3_fd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1835301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1835463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1835622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1835781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1835848Z ok (3.585s)
2022-11-23T02:20:42.1835853Z 
2022-11-23T02:20:42.1836048Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1836110Z Ran 1 test in 3.585s
2022-11-23T02:20:42.1836127Z 
2022-11-23T02:20:42.1836173Z OK
2022-11-23T02:20:42.1836177Z 
2022-11-23T02:20:42.1836260Z Generating XML reports...
2022-11-23T02:20:42.1836594Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020101.xml
2022-11-23T02:20:42.1836864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1836985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1837261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1837393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1837575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_azl6yt
2022-11-23T02:20:42.1837755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_azl6yt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1838006Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1838043Z 
2022-11-23T02:20:42.1838115Z Running tests...
2022-11-23T02:20:42.1838308Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1838529Z   test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11933
2022-11-23T02:20:42.1838685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11934
2022-11-23T02:20:42.1838838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11935
2022-11-23T02:20:42.1838989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11936
2022-11-23T02:20:42.1839277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1839397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1839678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1839809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1839990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7ivced7
2022-11-23T02:20:42.1840181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7ivced7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1840448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1840569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1840848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1840967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1841149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6i261ei
2022-11-23T02:20:42.1841340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6i261ei/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1841608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1841726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1842002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1842132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1842315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb3qvxsj
2022-11-23T02:20:42.1842494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb3qvxsj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1842760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1842879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1843156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1843287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1843466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qzsouw1
2022-11-23T02:20:42.1843656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qzsouw1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1843819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1843981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1844131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1844316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1844385Z ok (2.490s)
2022-11-23T02:20:42.1844390Z 
2022-11-23T02:20:42.1844584Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1844655Z Ran 1 test in 2.491s
2022-11-23T02:20:42.1844661Z 
2022-11-23T02:20:42.1844718Z OK
2022-11-23T02:20:42.1844723Z 
2022-11-23T02:20:42.1844802Z Generating XML reports...
2022-11-23T02:20:42.1845201Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020107.xml
2022-11-23T02:20:42.1845476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1845623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1845903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1846039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1846224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdvemof5
2022-11-23T02:20:42.1846419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdvemof5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1846676Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1846681Z 
2022-11-23T02:20:42.1846753Z Running tests...
2022-11-23T02:20:42.1846948Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1847166Z   test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12156
2022-11-23T02:20:42.1847323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12157
2022-11-23T02:20:42.1847473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12158
2022-11-23T02:20:42.1847625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12159
2022-11-23T02:20:42.1847898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1848017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1848295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1848427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1848608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxus5swe2
2022-11-23T02:20:42.1848790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxus5swe2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1849060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1849179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1849458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1849590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1849771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwckulr7s
2022-11-23T02:20:42.1850038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1850155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1850419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1850549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1850731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj676kzon
2022-11-23T02:20:42.1850952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwckulr7s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1851138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj676kzon/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1851408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1851531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1851806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1851967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1852136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq6zc36s
2022-11-23T02:20:42.1852326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq6zc36s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1852489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1852649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1852810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1852969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1853034Z ok (2.473s)
2022-11-23T02:20:42.1853039Z 
2022-11-23T02:20:42.1853232Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1853296Z Ran 1 test in 2.473s
2022-11-23T02:20:42.1853312Z 
2022-11-23T02:20:42.1853359Z OK
2022-11-23T02:20:42.1853364Z 
2022-11-23T02:20:42.1853446Z Generating XML reports...
2022-11-23T02:20:42.1853786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020111.xml
2022-11-23T02:20:42.1854059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1854180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1854460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1854593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1854775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofagboiz
2022-11-23T02:20:42.1855063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofagboiz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1855321Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1855327Z 
2022-11-23T02:20:42.1855397Z Running tests...
2022-11-23T02:20:42.1855590Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1855818Z   test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12379
2022-11-23T02:20:42.1855973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12380
2022-11-23T02:20:42.1856127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12381
2022-11-23T02:20:42.1856279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12382
2022-11-23T02:20:42.1856544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1856667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1856950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1857084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1857319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6naefl_
2022-11-23T02:20:42.1857517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6naefl_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1857789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1857911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1858192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1858313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1858536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd0zwp8r
2022-11-23T02:20:42.1858807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1858929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1859208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1859343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1859528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbdojcrz
2022-11-23T02:20:42.1859725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd0zwp8r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1859902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbdojcrz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1860174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1860294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1860570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1860705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1860886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24voraos
2022-11-23T02:20:42.1861079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24voraos/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1861242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1861401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1861549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1861710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1861774Z ok (2.174s)
2022-11-23T02:20:42.1861779Z 
2022-11-23T02:20:42.1861974Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1862047Z Ran 1 test in 2.174s
2022-11-23T02:20:42.1862054Z 
2022-11-23T02:20:42.1862111Z OK
2022-11-23T02:20:42.1862116Z 
2022-11-23T02:20:42.1862196Z Generating XML reports...
2022-11-23T02:20:42.1862534Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020116.xml
2022-11-23T02:20:42.1862804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1862916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1863193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1863328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1863512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb5a__b2
2022-11-23T02:20:42.1863730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb5a__b2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1863986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1863991Z 
2022-11-23T02:20:42.1864061Z Running tests...
2022-11-23T02:20:42.1864254Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1864464Z   test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12434
2022-11-23T02:20:42.1864618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12435
2022-11-23T02:20:42.1864797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12436
2022-11-23T02:20:42.1864947Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12437
2022-11-23T02:20:42.1865219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1865341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1865709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1865848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1866032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamzxunop
2022-11-23T02:20:42.1866291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1866410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1866691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1866824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1867006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk4febgg
2022-11-23T02:20:42.1867201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamzxunop/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1867392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk4febgg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1867663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1867772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1868049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1868184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1868365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9ynd2b7
2022-11-23T02:20:42.1868556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9ynd2b7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1868827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1868948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1869228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1869360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1869534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2d7ijvp
2022-11-23T02:20:42.1869728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2d7ijvp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1869891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1870053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1870248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1870409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1870474Z ok (2.475s)
2022-11-23T02:20:42.1870479Z 
2022-11-23T02:20:42.1870708Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1870796Z Ran 1 test in 2.476s
2022-11-23T02:20:42.1870801Z 
2022-11-23T02:20:42.1870861Z OK
2022-11-23T02:20:42.1870866Z 
2022-11-23T02:20:42.1870949Z Generating XML reports...
2022-11-23T02:20:42.1871291Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020120.xml
2022-11-23T02:20:42.1871599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1871724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1872006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1872142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1872325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wh59hgb
2022-11-23T02:20:42.1872505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wh59hgb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1872762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1872767Z 
2022-11-23T02:20:42.1872839Z Running tests...
2022-11-23T02:20:42.1873036Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1873260Z   test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12657
2022-11-23T02:20:42.1873414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12658
2022-11-23T02:20:42.1873568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12659
2022-11-23T02:20:42.1873718Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12660
2022-11-23T02:20:42.1873977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1874097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1874374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1874506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1874690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6__f2kk
2022-11-23T02:20:42.1874882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6__f2kk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1875154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1875274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1875555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1875676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1875858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvhfvtmn
2022-11-23T02:20:42.1876051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvhfvtmn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1876489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1876615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1876892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1877060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1877243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhwe9vn0
2022-11-23T02:20:42.1877424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhwe9vn0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1877696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1877816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1878135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1878268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1878448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzamvz5fh
2022-11-23T02:20:42.1878644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzamvz5fh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1878803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1878967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1879116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1879274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1879339Z ok (2.576s)
2022-11-23T02:20:42.1879345Z 
2022-11-23T02:20:42.1879540Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1879614Z Ran 1 test in 2.576s
2022-11-23T02:20:42.1879619Z 
2022-11-23T02:20:42.1879679Z OK
2022-11-23T02:20:42.1879683Z 
2022-11-23T02:20:42.1879766Z Generating XML reports...
2022-11-23T02:20:42.1880106Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020124.xml
2022-11-23T02:20:42.1880368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1880492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1880768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1880901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1881086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo6jdofy
2022-11-23T02:20:42.1881282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo6jdofy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1881537Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1881542Z 
2022-11-23T02:20:42.1881611Z Running tests...
2022-11-23T02:20:42.1881810Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1881923Z   test_call_method_on_rref (__main__.TensorPipeRpcTest)
2022-11-23T02:20:42.1882152Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12880
2022-11-23T02:20:42.1882306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12881
2022-11-23T02:20:42.1882456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12882
2022-11-23T02:20:42.1882604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12883
2022-11-23T02:20:42.1882878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1882997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1883305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1883438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1883610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_alt1tz
2022-11-23T02:20:42.1883803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_alt1tz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1884075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1884194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1884531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1884664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1884845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrs0w_2v
2022-11-23T02:20:42.1885103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrs0w_2v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1885364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1885484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1885762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1885894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1886076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv1yteg9
2022-11-23T02:20:42.1886352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1886474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1886751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1886885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1887053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n2t6oeu
2022-11-23T02:20:42.1887246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv1yteg9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1887432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n2t6oeu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1887596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1887761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1887922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1888081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1888148Z ok (2.581s)
2022-11-23T02:20:42.1888154Z 
2022-11-23T02:20:42.1888337Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1888412Z Ran 1 test in 2.582s
2022-11-23T02:20:42.1888417Z 
2022-11-23T02:20:42.1888476Z OK
2022-11-23T02:20:42.1888481Z 
2022-11-23T02:20:42.1888562Z Generating XML reports...
2022-11-23T02:20:42.1888901Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020128.xml
2022-11-23T02:20:42.1889173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1889295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1889573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1889706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1889908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkdg2mbk
2022-11-23T02:20:42.1890103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkdg2mbk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1890356Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1890361Z 
2022-11-23T02:20:42.1890432Z Running tests...
2022-11-23T02:20:42.1890626Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1890845Z   test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13103
2022-11-23T02:20:42.1891030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13104
2022-11-23T02:20:42.1891182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13105
2022-11-23T02:20:42.1891321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13106
2022-11-23T02:20:42.1891595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1891715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1891996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1892130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1892313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbavhxanw
2022-11-23T02:20:42.1892510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbavhxanw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1892784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1892903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1893170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1893304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1893486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8vr6nej
2022-11-23T02:20:42.1893677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8vr6nej/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1893946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1894068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1894348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1894481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1894654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcl907769
2022-11-23T02:20:42.1894844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcl907769/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1895230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1895350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1895625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1895757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1895942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3km3sbe
2022-11-23T02:20:42.1896134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3km3sbe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1896295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1896496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1896657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1896816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1896883Z ok (2.476s)
2022-11-23T02:20:42.1896888Z 
2022-11-23T02:20:42.1897084Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1897160Z Ran 1 test in 2.477s
2022-11-23T02:20:42.1897198Z 
2022-11-23T02:20:42.1897259Z OK
2022-11-23T02:20:42.1897263Z 
2022-11-23T02:20:42.1897346Z Generating XML reports...
2022-11-23T02:20:42.1897673Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020133.xml
2022-11-23T02:20:42.1897952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1898075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1898355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1898489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1898672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeh8iudvh
2022-11-23T02:20:42.1898866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeh8iudvh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1899124Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1899129Z 
2022-11-23T02:20:42.1899199Z Running tests...
2022-11-23T02:20:42.1899382Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1899602Z   test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13326
2022-11-23T02:20:42.1899759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13327
2022-11-23T02:20:42.1899914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13328
2022-11-23T02:20:42.1900069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13329
2022-11-23T02:20:42.1900341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1900461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1900744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1900864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1901046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yex4fu5
2022-11-23T02:20:42.1901241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yex4fu5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1901514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1901633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1901913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1902044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1902231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjnow07i
2022-11-23T02:20:42.1902422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjnow07i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1902680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1902835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1903114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1903246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1903425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrthd_p2
2022-11-23T02:20:42.1903616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrthd_p2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1903884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1904033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1904307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1904429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1904611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb6ql8bc
2022-11-23T02:20:42.1904801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb6ql8bc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1904964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1905127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1905284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1905445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1905511Z ok (2.574s)
2022-11-23T02:20:42.1905516Z 
2022-11-23T02:20:42.1905701Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1905773Z Ran 1 test in 2.574s
2022-11-23T02:20:42.1905778Z 
2022-11-23T02:20:42.1905835Z OK
2022-11-23T02:20:42.1905843Z 
2022-11-23T02:20:42.1905924Z Generating XML reports...
2022-11-23T02:20:42.1906262Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020137.xml
2022-11-23T02:20:42.1906534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1906656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1906935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1907069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1907240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9dev2vv
2022-11-23T02:20:42.1907430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9dev2vv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1907682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1907688Z 
2022-11-23T02:20:42.1907757Z Running tests...
2022-11-23T02:20:42.1907951Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1908168Z   test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13549
2022-11-23T02:20:42.1908323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13550
2022-11-23T02:20:42.1908474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13551
2022-11-23T02:20:42.1908616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13552
2022-11-23T02:20:42.1908890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1909013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1909321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1909459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1909641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnawo9_z
2022-11-23T02:20:42.1909839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnawo9_z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1910108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1910258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1910525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1910658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1910841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3_1syfz
2022-11-23T02:20:42.1911033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3_1syfz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1911303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1911426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1911707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1911840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1912010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8v4gxm02
2022-11-23T02:20:42.1912201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8v4gxm02/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1912471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1912592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1912868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1913000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1913181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfp1c63a7
2022-11-23T02:20:42.1913372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfp1c63a7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1913534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1913683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1913845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1914005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1914069Z ok (2.577s)
2022-11-23T02:20:42.1914075Z 
2022-11-23T02:20:42.1914271Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1914342Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1914347Z 
2022-11-23T02:20:42.1914406Z OK
2022-11-23T02:20:42.1914410Z 
2022-11-23T02:20:42.1914492Z Generating XML reports...
2022-11-23T02:20:42.1914822Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020142.xml
2022-11-23T02:20:42.1915095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1915214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1915492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1915655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1915840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp4w6btv
2022-11-23T02:20:42.1916034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp4w6btv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1916289Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1916294Z 
2022-11-23T02:20:42.1916364Z Running tests...
2022-11-23T02:20:42.1916546Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1916789Z   test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13772
2022-11-23T02:20:42.1916946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13773
2022-11-23T02:20:42.1917097Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13774
2022-11-23T02:20:42.1917250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13775
2022-11-23T02:20:42.1917524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1917642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1917923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1918045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1918231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj09k3ri9
2022-11-23T02:20:42.1918422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj09k3ri9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1918693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1918815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1919095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1919224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1919406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgsv2hvw
2022-11-23T02:20:42.1919597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgsv2hvw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1919852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1919975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1920252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1920386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1920571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni3lyc7o
2022-11-23T02:20:42.1920763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni3lyc7o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1921032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1921151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1921428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1921550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1921730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiguluz8
2022-11-23T02:20:42.1921921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiguluz8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1922112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1922274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1922434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1922593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1922683Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.1922790Z ValueError('Expected error')
2022-11-23T02:20:42.1922924Z Traceback (most recent call last):
2022-11-23T02:20:42.1923187Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1923323Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1923609Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1923698Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1923781Z ValueError: Expected error
2022-11-23T02:20:42.1923786Z 
2022-11-23T02:20:42.1923874Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1923979Z ValueError('Expected error')
2022-11-23T02:20:42.1924067Z Traceback (most recent call last):
2022-11-23T02:20:42.1924323Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1924453Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1924733Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1924822Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1924971Z ValueError: Expected error
2022-11-23T02:20:42.1924978Z 
2022-11-23T02:20:42.1925055Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1925174Z ValueError('Expected error')
2022-11-23T02:20:42.1925268Z Traceback (most recent call last):
2022-11-23T02:20:42.1925525Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1925657Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1925933Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1926022Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1926104Z ValueError: Expected error
2022-11-23T02:20:42.1926109Z 
2022-11-23T02:20:42.1926186Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.1926304Z ValueError('Expected error')
2022-11-23T02:20:42.1926391Z Traceback (most recent call last):
2022-11-23T02:20:42.1926643Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1926773Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1927047Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1927136Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1927217Z ValueError: Expected error
2022-11-23T02:20:42.1927222Z 
2022-11-23T02:20:42.1927276Z ok (2.578s)
2022-11-23T02:20:42.1927281Z 
2022-11-23T02:20:42.1927475Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1927548Z Ran 1 test in 2.578s
2022-11-23T02:20:42.1927553Z 
2022-11-23T02:20:42.1927611Z OK
2022-11-23T02:20:42.1927618Z 
2022-11-23T02:20:42.1927699Z Generating XML reports...
2022-11-23T02:20:42.1928040Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020146.xml
2022-11-23T02:20:42.1928314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1928484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1928752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1928885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1929068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1nzdenj
2022-11-23T02:20:42.1929262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1nzdenj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1929513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1929550Z 
2022-11-23T02:20:42.1929622Z Running tests...
2022-11-23T02:20:42.1929816Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1930035Z   test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13995
2022-11-23T02:20:42.1930192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13996
2022-11-23T02:20:42.1930334Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13997
2022-11-23T02:20:42.1930484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13998
2022-11-23T02:20:42.1930760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1930881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1931162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1931298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1931482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz548nj1
2022-11-23T02:20:42.1931678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz548nj1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1931951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1932062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1932340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1932473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1932654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_fdth7p
2022-11-23T02:20:42.1932849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_fdth7p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1933118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1933239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1933516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1933639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1933821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v2heze1
2022-11-23T02:20:42.1934016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v2heze1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1934284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1934405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1934680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1934813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1935158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgp5ixd5
2022-11-23T02:20:42.1935357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgp5ixd5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1935507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1935674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1935832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1935990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1936091Z ok (2.474s)
2022-11-23T02:20:42.1936095Z 
2022-11-23T02:20:42.1936292Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1936366Z Ran 1 test in 2.474s
2022-11-23T02:20:42.1936371Z 
2022-11-23T02:20:42.1936431Z OK
2022-11-23T02:20:42.1936435Z 
2022-11-23T02:20:42.1936507Z Generating XML reports...
2022-11-23T02:20:42.1936846Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020151.xml
2022-11-23T02:20:42.1937118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1937242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1937523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1937658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1937843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfw4844ss
2022-11-23T02:20:42.1938035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfw4844ss/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1938288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1938293Z 
2022-11-23T02:20:42.1938352Z Running tests...
2022-11-23T02:20:42.1938548Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1938769Z   test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14218
2022-11-23T02:20:42.1938924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14219
2022-11-23T02:20:42.1939075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14220
2022-11-23T02:20:42.1939225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14221
2022-11-23T02:20:42.1939495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1939616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1939888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1940021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1940202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0jnwxc7
2022-11-23T02:20:42.1940396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0jnwxc7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1940664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1940785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1941064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1941198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1941379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9v166q3
2022-11-23T02:20:42.1941592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9v166q3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1941861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1941981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1942258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1942391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1942602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfcqgq7i
2022-11-23T02:20:42.1942793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfcqgq7i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1943062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1943175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1943452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1943583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1943764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1k8gm0g
2022-11-23T02:20:42.1943957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1k8gm0g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1944119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1944280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1944444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1944605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1944683Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.1944801Z ValueError('Expected error')
2022-11-23T02:20:42.1944890Z Traceback (most recent call last):
2022-11-23T02:20:42.1945150Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1945285Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1945565Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1945658Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1945730Z ValueError: Expected error
2022-11-23T02:20:42.1945746Z 
2022-11-23T02:20:42.1945823Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.1945941Z ValueError('Expected error')
2022-11-23T02:20:42.1946029Z Traceback (most recent call last):
2022-11-23T02:20:42.1946284Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1946413Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1946690Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1946781Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1946851Z ValueError: Expected error
2022-11-23T02:20:42.1946867Z 
2022-11-23T02:20:42.1946941Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.1947056Z ValueError('Expected error')
2022-11-23T02:20:42.1947145Z Traceback (most recent call last):
2022-11-23T02:20:42.1947399Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1947527Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1947827Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1947924Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1948008Z ValueError: Expected error
2022-11-23T02:20:42.1948013Z 
2022-11-23T02:20:42.1948103Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1948207Z ValueError('Expected error')
2022-11-23T02:20:42.1948296Z Traceback (most recent call last):
2022-11-23T02:20:42.1948549Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1948679Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1948983Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.1949072Z     raise ValueError(expected_err)
2022-11-23T02:20:42.1949155Z ValueError: Expected error
2022-11-23T02:20:42.1949160Z 
2022-11-23T02:20:42.1949225Z ok (2.476s)
2022-11-23T02:20:42.1949230Z 
2022-11-23T02:20:42.1949415Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1949491Z Ran 1 test in 2.476s
2022-11-23T02:20:42.1949496Z 
2022-11-23T02:20:42.1949554Z OK
2022-11-23T02:20:42.1949559Z 
2022-11-23T02:20:42.1949640Z Generating XML reports...
2022-11-23T02:20:42.1949977Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020155.xml
2022-11-23T02:20:42.1950250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1950372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1950653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1950786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1950958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cr4by61
2022-11-23T02:20:42.1951158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cr4by61/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1951409Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1951414Z 
2022-11-23T02:20:42.1951486Z Running tests...
2022-11-23T02:20:42.1951678Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1951896Z   test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14441
2022-11-23T02:20:42.1952054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14442
2022-11-23T02:20:42.1952207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14443
2022-11-23T02:20:42.1952345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14444
2022-11-23T02:20:42.1952620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1952741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1953021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1953155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1953343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpooz3ns
2022-11-23T02:20:42.1953539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpooz3ns/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1953813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1953935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1954201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1954362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1954546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7abvky4b
2022-11-23T02:20:42.1954743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7abvky4b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1955013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1955135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1955448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1955581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1955751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg57aa1g4
2022-11-23T02:20:42.1955945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg57aa1g4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1956217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1956337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1956618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1956750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1956933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm1gbjeh
2022-11-23T02:20:42.1957130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm1gbjeh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1957293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1957448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1957610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1957769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1957837Z ok (2.475s)
2022-11-23T02:20:42.1957842Z 
2022-11-23T02:20:42.1958039Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1958112Z Ran 1 test in 2.475s
2022-11-23T02:20:42.1958117Z 
2022-11-23T02:20:42.1958174Z OK
2022-11-23T02:20:42.1958178Z 
2022-11-23T02:20:42.1958259Z Generating XML reports...
2022-11-23T02:20:42.1958589Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020159.xml
2022-11-23T02:20:42.1958862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1958985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1959266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1959402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1959587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ipgxmel
2022-11-23T02:20:42.1959782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ipgxmel/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1960033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1960040Z 
2022-11-23T02:20:42.1960110Z Running tests...
2022-11-23T02:20:42.1960292Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1960518Z   test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14664
2022-11-23T02:20:42.1960704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14665
2022-11-23T02:20:42.1960858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14666
2022-11-23T02:20:42.1961010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14667
2022-11-23T02:20:42.1961283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1961403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1961682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1961844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1962017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufbav3ee
2022-11-23T02:20:42.1962214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufbav3ee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1962486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1962606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1962886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1963018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1963200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xo8t_7u
2022-11-23T02:20:42.1963392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xo8t_7u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1963650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1963770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1964052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1964186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1964370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81zyz_8f
2022-11-23T02:20:42.1964561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81zyz_8f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1964830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1965020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1965302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1965422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1965605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz3w84ps
2022-11-23T02:20:42.1965796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz3w84ps/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1965958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1966119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1966278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1966437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1966505Z ok (2.480s)
2022-11-23T02:20:42.1966510Z 
2022-11-23T02:20:42.1966696Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1966769Z Ran 1 test in 2.480s
2022-11-23T02:20:42.1966773Z 
2022-11-23T02:20:42.1966831Z OK
2022-11-23T02:20:42.1966836Z 
2022-11-23T02:20:42.1966917Z Generating XML reports...
2022-11-23T02:20:42.1967290Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020204.xml
2022-11-23T02:20:42.1967566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1967688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1967968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1968099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1968314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkge9ih0x
2022-11-23T02:20:42.1968509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkge9ih0x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1968762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1968770Z 
2022-11-23T02:20:42.1968841Z Running tests...
2022-11-23T02:20:42.1969033Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1969258Z   test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14887
2022-11-23T02:20:42.1969413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14888
2022-11-23T02:20:42.1969568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14889
2022-11-23T02:20:42.1969706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14890
2022-11-23T02:20:42.1969982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1970103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1970387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1970523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1970705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e_m3b4b
2022-11-23T02:20:42.1970896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e_m3b4b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1971167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1971324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1971658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1971794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1971976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkbvnefj
2022-11-23T02:20:42.1972174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkbvnefj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1972444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1972563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1972841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1972972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1973145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla4ugr97
2022-11-23T02:20:42.1973337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla4ugr97/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1973606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1973766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1974046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1974180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1974364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvho0xaw
2022-11-23T02:20:42.1974557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvho0xaw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1974721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1974999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1975163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1975324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1975393Z ok (2.472s)
2022-11-23T02:20:42.1975400Z 
2022-11-23T02:20:42.1975597Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1975671Z Ran 1 test in 2.473s
2022-11-23T02:20:42.1975676Z 
2022-11-23T02:20:42.1975738Z OK
2022-11-23T02:20:42.1975743Z 
2022-11-23T02:20:42.1975826Z Generating XML reports...
2022-11-23T02:20:42.1976154Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020208.xml
2022-11-23T02:20:42.1976426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1976550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1976828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1976962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1977148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvgu0zza
2022-11-23T02:20:42.1977343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvgu0zza/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1977598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1977603Z 
2022-11-23T02:20:42.1977673Z Running tests...
2022-11-23T02:20:42.1977856Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1978091Z   test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15110
2022-11-23T02:20:42.1978248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15111
2022-11-23T02:20:42.1978400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15112
2022-11-23T02:20:42.1978556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15113
2022-11-23T02:20:42.1978828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1978949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1979227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1979362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1979533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26g2amjn
2022-11-23T02:20:42.1979729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26g2amjn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1980000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1980123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1980452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1980592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1980773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ubq1qa2
2022-11-23T02:20:42.1981043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1981151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1981461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1981595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1981775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_y2r_yj
2022-11-23T02:20:42.1981970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ubq1qa2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1982156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_y2r_yj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1982427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1982550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1982824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1982947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1983129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivmuu5ip
2022-11-23T02:20:42.1983324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivmuu5ip/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1983487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1983649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1983809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1983968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1984034Z ok (2.173s)
2022-11-23T02:20:42.1984039Z 
2022-11-23T02:20:42.1984223Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1984295Z Ran 1 test in 2.174s
2022-11-23T02:20:42.1984302Z 
2022-11-23T02:20:42.1984360Z OK
2022-11-23T02:20:42.1984365Z 
2022-11-23T02:20:42.1984448Z Generating XML reports...
2022-11-23T02:20:42.1984785Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020212.xml
2022-11-23T02:20:42.1985058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1985180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1985456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1985590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1985762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6d0vsm6
2022-11-23T02:20:42.1985955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6d0vsm6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1986212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.1986218Z 
2022-11-23T02:20:42.1986288Z Running tests...
2022-11-23T02:20:42.1986483Z ----------------------------------------------------------------------
2022-11-23T02:20:42.1986701Z   test_custom_exception_throw_during_reconstruction (__main__.TensorPipeRpcTest)
2022-11-23T02:20:42.1986936Z Test that we still throw info about the remote side exception even when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15165
2022-11-23T02:20:42.1987091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15166
2022-11-23T02:20:42.1987229Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15167
2022-11-23T02:20:42.1987383Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15168
2022-11-23T02:20:42.1987657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1987806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1988089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1988223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1988411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjucp5knm
2022-11-23T02:20:42.1988609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjucp5knm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1988877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1988986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1989265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1989399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1989581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam_lt63v
2022-11-23T02:20:42.1989771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam_lt63v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1990042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1990163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1990440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1990572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1990743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqncuks_q
2022-11-23T02:20:42.1990939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqncuks_q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1991207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.1991327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.1991607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.1991738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.1991919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3xdg842
2022-11-23T02:20:42.1992109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3xdg842/_remote_module_non_scriptable.py
2022-11-23T02:20:42.1992259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.1992419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.1992583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.1992743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.1992920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.1993128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.1993302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.1993473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.1993771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1994056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1994375Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1994665Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.1995083Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1995198Z CustomException('foo')
2022-11-23T02:20:42.1995292Z Traceback (most recent call last):
2022-11-23T02:20:42.1995552Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1995684Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1995979Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.1996067Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.1996227Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.1996233Z 
2022-11-23T02:20:42.1996237Z 
2022-11-23T02:20:42.1996297Z At:
2022-11-23T02:20:42.1996547Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2022-11-23T02:20:42.1996552Z 
2022-11-23T02:20:42.1996965Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1997076Z CustomException('foo')
2022-11-23T02:20:42.1997166Z Traceback (most recent call last):
2022-11-23T02:20:42.1997428Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1997563Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1997844Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.1997941Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.1998101Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.1998107Z 
2022-11-23T02:20:42.1998111Z 
2022-11-23T02:20:42.1998173Z At:
2022-11-23T02:20:42.1998420Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2022-11-23T02:20:42.1998425Z 
2022-11-23T02:20:42.1998515Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1998626Z CustomException('foo')
2022-11-23T02:20:42.1998715Z Traceback (most recent call last):
2022-11-23T02:20:42.1998962Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.1999097Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.1999389Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.1999486Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.1999645Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.1999680Z 
2022-11-23T02:20:42.1999771Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.1999881Z CustomException('foo')
2022-11-23T02:20:42.1999969Z Traceback (most recent call last):
2022-11-23T02:20:42.2000210Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2000339Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2000626Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.2000750Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.2000909Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.2000914Z 
2022-11-23T02:20:42.2001002Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2001111Z CustomException('foo')
2022-11-23T02:20:42.2001198Z Traceback (most recent call last):
2022-11-23T02:20:42.2001439Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2001568Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2001857Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.2001955Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.2002112Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.2002116Z 
2022-11-23T02:20:42.2002527Z Got msg RuntimeError: Failed to create original exception type. Error msg was __init__() missing 1 required positional argument: 'msg' Original exception on remote side was On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2002641Z CustomException('foo')
2022-11-23T02:20:42.2002735Z Traceback (most recent call last):
2022-11-23T02:20:42.2002995Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2003114Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2003408Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 409, in custom_raise_func
2022-11-23T02:20:42.2003504Z     raise CustomException(True, "foo")
2022-11-23T02:20:42.2003661Z torch.testing._internal.distributed.rpc.rpc_test.CustomException: foo
2022-11-23T02:20:42.2003666Z 
2022-11-23T02:20:42.2003671Z 
2022-11-23T02:20:42.2003731Z At:
2022-11-23T02:20:42.2003976Z   /opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(231): _handle_exception
2022-11-23T02:20:42.2003983Z 
2022-11-23T02:20:42.2004050Z ok (2.577s)
2022-11-23T02:20:42.2004055Z 
2022-11-23T02:20:42.2004249Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2004323Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2004328Z 
2022-11-23T02:20:42.2004373Z OK
2022-11-23T02:20:42.2004378Z 
2022-11-23T02:20:42.2004461Z Generating XML reports...
2022-11-23T02:20:42.2004799Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020216.xml
2022-11-23T02:20:42.2005145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2005268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2005554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2005692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2005879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr08kq6eu
2022-11-23T02:20:42.2006063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr08kq6eu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2006361Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2006367Z 
2022-11-23T02:20:42.2006439Z Running tests...
2022-11-23T02:20:42.2006632Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2006843Z   test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15400
2022-11-23T02:20:42.2006998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15401
2022-11-23T02:20:42.2007151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15402
2022-11-23T02:20:42.2007333Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15403
2022-11-23T02:20:42.2007608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2007718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2008001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2008134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2008318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dt2fbwr
2022-11-23T02:20:42.2008514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dt2fbwr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2008785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2008907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2009190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2009325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2009497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lmes2ow
2022-11-23T02:20:42.2009692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lmes2ow/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2009960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2010081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2010361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2010493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2010677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicfxq2xf
2022-11-23T02:20:42.2010866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicfxq2xf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2011124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2011247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2011525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2011659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2011841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprofvjgvt
2022-11-23T02:20:42.2012034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprofvjgvt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2012199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2012361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2012522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2012697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2012873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2013051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2013223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2013397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2013696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2014032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2014321Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2014612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2014667Z ok (3.577s)
2022-11-23T02:20:42.2014672Z 
2022-11-23T02:20:42.2014995Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2015071Z Ran 1 test in 3.577s
2022-11-23T02:20:42.2015075Z 
2022-11-23T02:20:42.2015136Z OK
2022-11-23T02:20:42.2015141Z 
2022-11-23T02:20:42.2015222Z Generating XML reports...
2022-11-23T02:20:42.2015563Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020221.xml
2022-11-23T02:20:42.2015839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2015963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2016231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2016366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2016550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0csdx89
2022-11-23T02:20:42.2016743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0csdx89/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2016997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2017002Z 
2022-11-23T02:20:42.2017075Z Running tests...
2022-11-23T02:20:42.2017274Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2017488Z   test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15635
2022-11-23T02:20:42.2017642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15636
2022-11-23T02:20:42.2017786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15637
2022-11-23T02:20:42.2017937Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15638
2022-11-23T02:20:42.2018210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2018331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2018611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2018745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2018930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2csruj4y
2022-11-23T02:20:42.2019124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2csruj4y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2019431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2019554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2019834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2019967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2020152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplj5o2enp
2022-11-23T02:20:42.2020343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplj5o2enp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2020654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2020776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2021055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2021177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2021359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqytaxjv8
2022-11-23T02:20:42.2021552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqytaxjv8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2021819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2021939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2022214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2022346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2022528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafa09hnn
2022-11-23T02:20:42.2022719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafa09hnn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2022869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2023032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2023192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2023352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2023417Z ok (2.574s)
2022-11-23T02:20:42.2023422Z 
2022-11-23T02:20:42.2023620Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2023693Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2023698Z 
2022-11-23T02:20:42.2023758Z OK
2022-11-23T02:20:42.2023762Z 
2022-11-23T02:20:42.2023831Z Generating XML reports...
2022-11-23T02:20:42.2024169Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020226.xml
2022-11-23T02:20:42.2024441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2024562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2024839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2024970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2025155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv2t2x_i
2022-11-23T02:20:42.2025349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv2t2x_i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2025602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2025607Z 
2022-11-23T02:20:42.2025665Z Running tests...
2022-11-23T02:20:42.2025888Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2026015Z   test_default_timeout_used (__main__.TensorPipeRpcTest)
2022-11-23T02:20:42.2026246Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15862
2022-11-23T02:20:42.2026400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15863
2022-11-23T02:20:42.2026553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15864
2022-11-23T02:20:42.2026703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15865
2022-11-23T02:20:42.2027002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2027112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2027394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2027529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2027713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9r9rszzs
2022-11-23T02:20:42.2027908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9r9rszzs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2028182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2028305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2028585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2028719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2028890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x63u59o
2022-11-23T02:20:42.2029161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2029284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2029563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2029695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2029877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_izpbvf
2022-11-23T02:20:42.2030070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x63u59o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2030260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_izpbvf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2030519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2030642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2030923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2031057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2031244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr8jp9qg
2022-11-23T02:20:42.2031436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr8jp9qg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2031597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2031760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2031920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2032068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2032163Z ok (4.491s)
2022-11-23T02:20:42.2032168Z 
2022-11-23T02:20:42.2032363Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2032436Z Ran 1 test in 4.492s
2022-11-23T02:20:42.2032441Z 
2022-11-23T02:20:42.2032504Z OK
2022-11-23T02:20:42.2032509Z 
2022-11-23T02:20:42.2032590Z Generating XML reports...
2022-11-23T02:20:42.2032928Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020231.xml
2022-11-23T02:20:42.2033198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2033340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2033621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2033756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2033945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpype432hv
2022-11-23T02:20:42.2034140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpype432hv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2034395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2034401Z 
2022-11-23T02:20:42.2034471Z Running tests...
2022-11-23T02:20:42.2034665Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2034886Z   test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16085
2022-11-23T02:20:42.2035032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16086
2022-11-23T02:20:42.2035185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16087
2022-11-23T02:20:42.2035338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16088
2022-11-23T02:20:42.2035611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2035732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2036013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2036146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2036327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaacewse2
2022-11-23T02:20:42.2036511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaacewse2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2036782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2036903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2037279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2037413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2037595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfu8d8nmt
2022-11-23T02:20:42.2037788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfu8d8nmt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2038060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2038183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2038448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2038580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2038795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lis42zx
2022-11-23T02:20:42.2038989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lis42zx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2039259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2039380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2039656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2039788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2039987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68x694_2
2022-11-23T02:20:42.2040176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68x694_2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2040339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2040504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2040667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2040828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2040893Z ok (2.573s)
2022-11-23T02:20:42.2040898Z 
2022-11-23T02:20:42.2041096Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2041169Z Ran 1 test in 2.573s
2022-11-23T02:20:42.2041176Z 
2022-11-23T02:20:42.2041223Z OK
2022-11-23T02:20:42.2041228Z 
2022-11-23T02:20:42.2041310Z Generating XML reports...
2022-11-23T02:20:42.2041649Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020237.xml
2022-11-23T02:20:42.2041923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2042048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2042323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2042455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2042636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5i2ha1ga
2022-11-23T02:20:42.2042815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5i2ha1ga/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2043071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2043076Z 
2022-11-23T02:20:42.2043149Z Running tests...
2022-11-23T02:20:42.2043344Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2043569Z   test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16308
2022-11-23T02:20:42.2043725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16309
2022-11-23T02:20:42.2043876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16310
2022-11-23T02:20:42.2044027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16311
2022-11-23T02:20:42.2044321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2044466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2044756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2044890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2045134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvox003sf
2022-11-23T02:20:42.2045364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvox003sf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2045641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2045764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2046046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2046179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2046381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmq4229z
2022-11-23T02:20:42.2046576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmq4229z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2046848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2046972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2047253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2047386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2047566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m3cyab9
2022-11-23T02:20:42.2047850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m3cyab9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2048175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2048302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2048581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2048717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2048900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8ed_17y
2022-11-23T02:20:42.2049090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8ed_17y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2049254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2049416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2049577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2049726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2049792Z ok (2.474s)
2022-11-23T02:20:42.2049797Z 
2022-11-23T02:20:42.2049994Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2050074Z Ran 1 test in 2.474s
2022-11-23T02:20:42.2050079Z 
2022-11-23T02:20:42.2050141Z OK
2022-11-23T02:20:42.2050148Z 
2022-11-23T02:20:42.2050230Z Generating XML reports...
2022-11-23T02:20:42.2050568Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020241.xml
2022-11-23T02:20:42.2050840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2050950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2051230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2051370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2051553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrxpz707
2022-11-23T02:20:42.2051747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrxpz707/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2052044Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2052049Z 
2022-11-23T02:20:42.2052121Z Running tests...
2022-11-23T02:20:42.2052317Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2052534Z   test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16531
2022-11-23T02:20:42.2052678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16532
2022-11-23T02:20:42.2052831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16533
2022-11-23T02:20:42.2053014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16534
2022-11-23T02:20:42.2053288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2053410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2053692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2053826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2054010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc25a2tj0
2022-11-23T02:20:42.2054190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc25a2tj0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2054458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2054580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2054964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2055099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2055283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71zkc23i
2022-11-23T02:20:42.2055472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71zkc23i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2055746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2055867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2056131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2056266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2056446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxngd2kan
2022-11-23T02:20:42.2056639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxngd2kan/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2056909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2057029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2057306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2057437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2057606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87rrodry
2022-11-23T02:20:42.2057798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87rrodry/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2057960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2058120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2058281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2058493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2058561Z ok (2.375s)
2022-11-23T02:20:42.2058566Z 
2022-11-23T02:20:42.2058761Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2058833Z Ran 1 test in 2.375s
2022-11-23T02:20:42.2058838Z 
2022-11-23T02:20:42.2058885Z OK
2022-11-23T02:20:42.2058889Z 
2022-11-23T02:20:42.2058971Z Generating XML reports...
2022-11-23T02:20:42.2059314Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020246.xml
2022-11-23T02:20:42.2059644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2059765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2060043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2060179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2060362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjjl0pcb
2022-11-23T02:20:42.2060545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjjl0pcb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2060798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2060803Z 
2022-11-23T02:20:42.2060874Z Running tests...
2022-11-23T02:20:42.2061069Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2061293Z   test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16666
2022-11-23T02:20:42.2061450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16667
2022-11-23T02:20:42.2061606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16668
2022-11-23T02:20:42.2061761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16669
2022-11-23T02:20:42.2062037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2062147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2062428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2062559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2062746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibvqxsf0
2022-11-23T02:20:42.2062940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibvqxsf0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2063213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2063335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2063613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2063744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2063915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6le3srem
2022-11-23T02:20:42.2064106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6le3srem/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2064374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2064496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2064771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2064934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2065117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasrekeji
2022-11-23T02:20:42.2065311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasrekeji/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2065570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2065692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2065969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2066130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2066315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wgn2x4w
2022-11-23T02:20:42.2066508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wgn2x4w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2066672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2066833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2066995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2067143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2067207Z ok (2.372s)
2022-11-23T02:20:42.2067212Z 
2022-11-23T02:20:42.2067407Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2067483Z Ran 1 test in 2.373s
2022-11-23T02:20:42.2067488Z 
2022-11-23T02:20:42.2067547Z OK
2022-11-23T02:20:42.2067552Z 
2022-11-23T02:20:42.2067633Z Generating XML reports...
2022-11-23T02:20:42.2067971Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020250.xml
2022-11-23T02:20:42.2068245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2068354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2068634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2068769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2068953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy9ok_8v
2022-11-23T02:20:42.2069151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy9ok_8v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2069406Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2069411Z 
2022-11-23T02:20:42.2069482Z Running tests...
2022-11-23T02:20:42.2069676Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2069893Z   test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16801
2022-11-23T02:20:42.2070035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16802
2022-11-23T02:20:42.2070188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16803
2022-11-23T02:20:42.2070337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16804
2022-11-23T02:20:42.2070611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2070734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2071016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2071150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2071362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lhx2ipn
2022-11-23T02:20:42.2071546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lhx2ipn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2071817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2071937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2072219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2072380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2072564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhb47ktf
2022-11-23T02:20:42.2072757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhb47ktf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2073027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2073148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2073413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2073544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2073726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3kthp98
2022-11-23T02:20:42.2073919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3kthp98/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2074190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2074310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2074590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2074721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2074891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffot90bk
2022-11-23T02:20:42.2075085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffot90bk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2075247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2075409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2075569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2075729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2075793Z ok (2.577s)
2022-11-23T02:20:42.2075798Z 
2022-11-23T02:20:42.2075989Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2076065Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2076070Z 
2022-11-23T02:20:42.2076117Z OK
2022-11-23T02:20:42.2076121Z 
2022-11-23T02:20:42.2076202Z Generating XML reports...
2022-11-23T02:20:42.2076540Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020254.xml
2022-11-23T02:20:42.2076807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2076927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2077205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2077338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2077521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj28u_nv
2022-11-23T02:20:42.2077733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj28u_nv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2077988Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2077993Z 
2022-11-23T02:20:42.2078064Z Running tests...
2022-11-23T02:20:42.2078258Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2078482Z   test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17024
2022-11-23T02:20:42.2078638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17025
2022-11-23T02:20:42.2078819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17026
2022-11-23T02:20:42.2078971Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17027
2022-11-23T02:20:42.2079243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2079354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2079636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2079767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2079948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb3qozfy
2022-11-23T02:20:42.2080144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb3qozfy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2080419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2080543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2080822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2080957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2081141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv9pqs_c
2022-11-23T02:20:42.2081334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv9pqs_c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2081604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2081723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2082000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2082134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2082316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtnkgfx3
2022-11-23T02:20:42.2082510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtnkgfx3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2082767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2082887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2083164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2083295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2083477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10mbbdao
2022-11-23T02:20:42.2083669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10mbbdao/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2083831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2083997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2084197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2084349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2084440Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2085177Z AttributeError("Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize\n            function code. Ensure that UDFs are defined on both caller and\n            callee modules.")
2022-11-23T02:20:42.2085338Z Traceback (most recent call last):
2022-11-23T02:20:42.2085601Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 160, in deserialize
2022-11-23T02:20:42.2085687Z     ret = unpickler.load()
2022-11-23T02:20:42.2086116Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'>
2022-11-23T02:20:42.2086122Z 
2022-11-23T02:20:42.2086257Z The above exception was the direct cause of the following exception:
2022-11-23T02:20:42.2086262Z 
2022-11-23T02:20:42.2086352Z Traceback (most recent call last):
2022-11-23T02:20:42.2086600Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-11-23T02:20:42.2086674Z     raise python_udf
2022-11-23T02:20:42.2087157Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize
2022-11-23T02:20:42.2087292Z             function code. Ensure that UDFs are defined on both caller and
2022-11-23T02:20:42.2087367Z             callee modules.
2022-11-23T02:20:42.2087372Z 
2022-11-23T02:20:42.2087440Z ok (2.479s)
2022-11-23T02:20:42.2087445Z 
2022-11-23T02:20:42.2087640Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2087713Z Ran 1 test in 2.480s
2022-11-23T02:20:42.2087717Z 
2022-11-23T02:20:42.2087776Z OK
2022-11-23T02:20:42.2087781Z 
2022-11-23T02:20:42.2087850Z Generating XML reports...
2022-11-23T02:20:42.2088187Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020259.xml
2022-11-23T02:20:42.2088460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2088584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2088866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2088998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2089184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpls2il73g
2022-11-23T02:20:42.2089378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpls2il73g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2089630Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2089635Z 
2022-11-23T02:20:42.2089694Z Running tests...
2022-11-23T02:20:42.2089886Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2090100Z   test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17247
2022-11-23T02:20:42.2090259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17248
2022-11-23T02:20:42.2090414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17249
2022-11-23T02:20:42.2090567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17250
2022-11-23T02:20:42.2090871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2090994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2091262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2091396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2091580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qz1iv1b
2022-11-23T02:20:42.2091801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qz1iv1b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2092074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2092195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2092474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2092607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2092789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav2daczu
2022-11-23T02:20:42.2092969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav2daczu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2093237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2093362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2093639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2093770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2093953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob_nh3z8
2022-11-23T02:20:42.2094143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob_nh3z8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2094410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2094531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2094795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2095035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2095221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mc61oy8
2022-11-23T02:20:42.2095411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mc61oy8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2095572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2095736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2095895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2096056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2096108Z ok (2.574s)
2022-11-23T02:20:42.2096114Z 
2022-11-23T02:20:42.2096310Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2096384Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2096389Z 
2022-11-23T02:20:42.2096450Z OK
2022-11-23T02:20:42.2096454Z 
2022-11-23T02:20:42.2096536Z Generating XML reports...
2022-11-23T02:20:42.2096874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020303.xml
2022-11-23T02:20:42.2097147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2097322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2097603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2097724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2097908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseopmpjt
2022-11-23T02:20:42.2098102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseopmpjt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2098390Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2098395Z 
2022-11-23T02:20:42.2098465Z Running tests...
2022-11-23T02:20:42.2098662Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2098890Z   test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17470
2022-11-23T02:20:42.2099045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17471
2022-11-23T02:20:42.2099187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17472
2022-11-23T02:20:42.2099338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17473
2022-11-23T02:20:42.2099611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2099732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2100018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2100150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2100333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4o_q_j5y
2022-11-23T02:20:42.2100529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4o_q_j5y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2100800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2100909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2101190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2101323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2101513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwcsg4rf
2022-11-23T02:20:42.2101707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwcsg4rf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2101976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2102098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2102375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2102496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2102677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgvv4yq0
2022-11-23T02:20:42.2102946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2103067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2103346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2103479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2103662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jgwnhka
2022-11-23T02:20:42.2103886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgvv4yq0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2104077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jgwnhka/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2104227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2104389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2104551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2104756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2104847Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2104965Z ValueError('Expected error')
2022-11-23T02:20:42.2105056Z Traceback (most recent call last):
2022-11-23T02:20:42.2105318Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2105440Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2105723Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2105813Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2105896Z ValueError: Expected error
2022-11-23T02:20:42.2105901Z 
2022-11-23T02:20:42.2105989Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2106106Z ValueError('Expected error')
2022-11-23T02:20:42.2106195Z Traceback (most recent call last):
2022-11-23T02:20:42.2106439Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2106570Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2106848Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2106939Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2107021Z ValueError: Expected error
2022-11-23T02:20:42.2107026Z 
2022-11-23T02:20:42.2107112Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2107229Z ValueError('Expected error')
2022-11-23T02:20:42.2107317Z Traceback (most recent call last):
2022-11-23T02:20:42.2107560Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2107689Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2107962Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2108054Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2108138Z ValueError: Expected error
2022-11-23T02:20:42.2108143Z 
2022-11-23T02:20:42.2108230Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2108347Z ValueError('Expected error')
2022-11-23T02:20:42.2108438Z Traceback (most recent call last):
2022-11-23T02:20:42.2108677Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2108804Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2109077Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2109167Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2109250Z ValueError: Expected error
2022-11-23T02:20:42.2109255Z 
2022-11-23T02:20:42.2109319Z ok (2.474s)
2022-11-23T02:20:42.2109326Z 
2022-11-23T02:20:42.2109522Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2109595Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2109601Z 
2022-11-23T02:20:42.2109646Z OK
2022-11-23T02:20:42.2109653Z 
2022-11-23T02:20:42.2109736Z Generating XML reports...
2022-11-23T02:20:42.2110113Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020307.xml
2022-11-23T02:20:42.2110389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2110512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2110791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2110925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2111112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2hjnwlo
2022-11-23T02:20:42.2111349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2hjnwlo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2111590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2111595Z 
2022-11-23T02:20:42.2111667Z Running tests...
2022-11-23T02:20:42.2111863Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2112080Z   test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17693
2022-11-23T02:20:42.2112236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17694
2022-11-23T02:20:42.2112388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17695
2022-11-23T02:20:42.2112540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17696
2022-11-23T02:20:42.2112816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2112925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2113205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2113341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2113528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo4vat09
2022-11-23T02:20:42.2113722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo4vat09/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2113991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2114113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2114393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2114527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2114698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtsoh295
2022-11-23T02:20:42.2114891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtsoh295/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2115161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2115282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2115559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2115692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2115873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzifeubz8
2022-11-23T02:20:42.2116067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzifeubz8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2116335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2116442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2116749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2116885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2117071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnv8krvt
2022-11-23T02:20:42.2117265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnv8krvt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2117426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2117615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2117774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2117920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2117984Z ok (2.478s)
2022-11-23T02:20:42.2117989Z 
2022-11-23T02:20:42.2118184Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2118257Z Ran 1 test in 2.479s
2022-11-23T02:20:42.2118262Z 
2022-11-23T02:20:42.2118323Z OK
2022-11-23T02:20:42.2118328Z 
2022-11-23T02:20:42.2118410Z Generating XML reports...
2022-11-23T02:20:42.2118744Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020312.xml
2022-11-23T02:20:42.2119016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2119127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2119411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2119545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2119733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajmjnt78
2022-11-23T02:20:42.2119927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajmjnt78/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2120180Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2120185Z 
2022-11-23T02:20:42.2120255Z Running tests...
2022-11-23T02:20:42.2120448Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2120674Z   test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17916
2022-11-23T02:20:42.2120819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17917
2022-11-23T02:20:42.2120973Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17918
2022-11-23T02:20:42.2121124Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17919
2022-11-23T02:20:42.2121398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2121517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2121796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2121929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2122110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_ic5vww
2022-11-23T02:20:42.2122300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_ic5vww/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2122560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2122680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2122998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2123133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2123315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbai7ydv
2022-11-23T02:20:42.2123509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbai7ydv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2123778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2123899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2124190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2124322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2124502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkq4d5qv
2022-11-23T02:20:42.2124696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkq4d5qv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2125064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2125189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2125472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2125604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2125789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzh3m1hf
2022-11-23T02:20:42.2125972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzh3m1hf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2126133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2126297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2126458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2126616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2126682Z ok (2.475s)
2022-11-23T02:20:42.2126687Z 
2022-11-23T02:20:42.2126883Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2126957Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2126962Z 
2022-11-23T02:20:42.2127009Z OK
2022-11-23T02:20:42.2127015Z 
2022-11-23T02:20:42.2127098Z Generating XML reports...
2022-11-23T02:20:42.2127436Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020316.xml
2022-11-23T02:20:42.2127707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2127832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2128108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2128241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2128427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilfjpc20
2022-11-23T02:20:42.2128619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilfjpc20/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2128861Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2128872Z 
2022-11-23T02:20:42.2128944Z Running tests...
2022-11-23T02:20:42.2129138Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2129854Z   test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.251s)
2022-11-23T02:20:42.2129861Z 
2022-11-23T02:20:42.2130055Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2130130Z Ran 1 test in 0.251s
2022-11-23T02:20:42.2130135Z 
2022-11-23T02:20:42.2130205Z OK (skipped=1)
2022-11-23T02:20:42.2130210Z 
2022-11-23T02:20:42.2130290Z Generating XML reports...
2022-11-23T02:20:42.2130622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020320.xml
2022-11-23T02:20:42.2130921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2131031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2131309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2131443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2131627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthg28mrf
2022-11-23T02:20:42.2131821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthg28mrf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2132072Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2132077Z 
2022-11-23T02:20:42.2132150Z Running tests...
2022-11-23T02:20:42.2132343Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2132549Z   test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18149
2022-11-23T02:20:42.2132703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18150
2022-11-23T02:20:42.2132856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18151
2022-11-23T02:20:42.2133009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18152
2022-11-23T02:20:42.2133285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2133408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2133690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2133824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2134009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1f5qe9vx
2022-11-23T02:20:42.2134191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1f5qe9vx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2134468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2134589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2134982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2135121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2135305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppksj49ro
2022-11-23T02:20:42.2135502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppksj49ro/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2135774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2135883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2136215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2136348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2136529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcf439qe
2022-11-23T02:20:42.2136721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcf439qe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2136993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2137114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2137428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2137561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2137732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1takrfde
2022-11-23T02:20:42.2137928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1takrfde/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2138089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2138250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2138411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2138571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2138637Z ok (2.578s)
2022-11-23T02:20:42.2138644Z 
2022-11-23T02:20:42.2138842Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2138903Z Ran 1 test in 2.578s
2022-11-23T02:20:42.2138921Z 
2022-11-23T02:20:42.2138967Z OK
2022-11-23T02:20:42.2138972Z 
2022-11-23T02:20:42.2139052Z Generating XML reports...
2022-11-23T02:20:42.2139394Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020322.xml
2022-11-23T02:20:42.2139666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2139790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2140069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2140202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2140385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfwwf7x9
2022-11-23T02:20:42.2140569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfwwf7x9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2140824Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2140829Z 
2022-11-23T02:20:42.2140901Z Running tests...
2022-11-23T02:20:42.2141096Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2141244Z   test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2022-11-23T02:20:42.2141434Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18372
2022-11-23T02:20:42.2141588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18373
2022-11-23T02:20:42.2141740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18374
2022-11-23T02:20:42.2141878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18375
2022-11-23T02:20:42.2142154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2142274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2142586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2142720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2142902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9trfhyu
2022-11-23T02:20:42.2143096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9trfhyu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2143363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2143484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2143794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2143926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2144109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xfq1p8i
2022-11-23T02:20:42.2144304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xfq1p8i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2144570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2144690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2145031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2145197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2145370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt6vtqxs
2022-11-23T02:20:42.2145562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt6vtqxs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2145835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2145957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2146234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2146364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2146544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxbxzugz
2022-11-23T02:20:42.2146735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxbxzugz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2146895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2147048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2147205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2147365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2147433Z ok (2.675s)
2022-11-23T02:20:42.2147438Z 
2022-11-23T02:20:42.2147635Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2147707Z Ran 1 test in 2.675s
2022-11-23T02:20:42.2147712Z 
2022-11-23T02:20:42.2147769Z OK
2022-11-23T02:20:42.2147774Z 
2022-11-23T02:20:42.2147857Z Generating XML reports...
2022-11-23T02:20:42.2148181Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020327.xml
2022-11-23T02:20:42.2148452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2148574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2148853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2148986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2149204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcgb4dzm
2022-11-23T02:20:42.2149400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcgb4dzm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2149652Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2149658Z 
2022-11-23T02:20:42.2149728Z Running tests...
2022-11-23T02:20:42.2149912Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2150142Z   test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18595
2022-11-23T02:20:42.2150345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18596
2022-11-23T02:20:42.2150499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18597
2022-11-23T02:20:42.2150652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18598
2022-11-23T02:20:42.2150925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2151046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2151328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2151462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2151634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rz0y70n
2022-11-23T02:20:42.2151831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rz0y70n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2152103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2152224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2152504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2152637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2152821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50eepe70
2022-11-23T02:20:42.2153011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50eepe70/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2153269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2153392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2153672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2153802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2153986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfak2uzxe
2022-11-23T02:20:42.2154177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfak2uzxe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2154448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2154568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2154845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2154968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2155151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpab7m8pvh
2022-11-23T02:20:42.2155346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpab7m8pvh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2155540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2155701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2155861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2156021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2156197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2156358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2156561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2156731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2157038Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2157341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2157635Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2157927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2158192Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2158456Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2158717Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2158974Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2159257Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2022-11-23T02:20:42.2159519Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2159777Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2160070Z [W tensorpipe_agent.cpp:915] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2022-11-23T02:20:42.2160139Z ok (2.576s)
2022-11-23T02:20:42.2160144Z 
2022-11-23T02:20:42.2160341Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2160415Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2160420Z 
2022-11-23T02:20:42.2160478Z OK
2022-11-23T02:20:42.2160483Z 
2022-11-23T02:20:42.2160566Z Generating XML reports...
2022-11-23T02:20:42.2160913Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020331.xml
2022-11-23T02:20:42.2161282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2161442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2161859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2162085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2162440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkpovk0u
2022-11-23T02:20:42.2162768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkpovk0u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2163031Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2163071Z 
2022-11-23T02:20:42.2163163Z Running tests...
2022-11-23T02:20:42.2163360Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2163568Z   test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18830
2022-11-23T02:20:42.2163728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18831
2022-11-23T02:20:42.2163885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18832
2022-11-23T02:20:42.2164039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18833
2022-11-23T02:20:42.2164318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2164441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2164724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2164860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2165112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeboheh3f
2022-11-23T02:20:42.2165296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeboheh3f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2165570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2165695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2165980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2166115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2166297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6waknzv5
2022-11-23T02:20:42.2166497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6waknzv5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2166772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2166881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2167163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2167300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2167488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnh6d3xs
2022-11-23T02:20:42.2167686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnh6d3xs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2167954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2168075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2168387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2168519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2168698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp356vn_4z
2022-11-23T02:20:42.2168907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp356vn_4z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2169068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2169230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2169391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2169551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2169757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2169932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2170105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2170278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2170564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2170860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2171150Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2171437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2171992Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.2172181Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2022-11-23T02:20:42.2172186Z 
2022-11-23T02:20:42.2172718Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.2172903Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2022-11-23T02:20:42.2172908Z 
2022-11-23T02:20:42.2173430Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.2173606Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2022-11-23T02:20:42.2173614Z 
2022-11-23T02:20:42.2174164Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.2174342Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2022-11-23T02:20:42.2174347Z 
2022-11-23T02:20:42.2174413Z ok (2.684s)
2022-11-23T02:20:42.2174417Z 
2022-11-23T02:20:42.2174615Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2174687Z Ran 1 test in 2.684s
2022-11-23T02:20:42.2174719Z 
2022-11-23T02:20:42.2174778Z OK
2022-11-23T02:20:42.2174782Z 
2022-11-23T02:20:42.2174972Z Generating XML reports...
2022-11-23T02:20:42.2175306Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020336.xml
2022-11-23T02:20:42.2175582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2175705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2175985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2176119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2176303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xfc9c71
2022-11-23T02:20:42.2176497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xfc9c71/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2176751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2176755Z 
2022-11-23T02:20:42.2176826Z Running tests...
2022-11-23T02:20:42.2177009Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2177228Z   test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19065
2022-11-23T02:20:42.2177383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19066
2022-11-23T02:20:42.2177536Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19067
2022-11-23T02:20:42.2177688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19068
2022-11-23T02:20:42.2177965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2178087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2178369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2178492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2178676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61vqmbet
2022-11-23T02:20:42.2178870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61vqmbet/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2179143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2179264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2179545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2179677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2179861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u5m5fuc
2022-11-23T02:20:42.2180053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u5m5fuc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2180313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2180490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2180770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2180903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2181086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dh7z7sn
2022-11-23T02:20:42.2181279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dh7z7sn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2181587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2181712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2181979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2182114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2182297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1kxv3js
2022-11-23T02:20:42.2182489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1kxv3js/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2182651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2182813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2182975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2183137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2183314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2183477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2183651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2183822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2184123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2184425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2184716Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2185008Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2185073Z ok (2.675s)
2022-11-23T02:20:42.2185078Z 
2022-11-23T02:20:42.2185273Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2185335Z Ran 1 test in 2.675s
2022-11-23T02:20:42.2185339Z 
2022-11-23T02:20:42.2185397Z OK
2022-11-23T02:20:42.2185402Z 
2022-11-23T02:20:42.2185482Z Generating XML reports...
2022-11-23T02:20:42.2185820Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020340.xml
2022-11-23T02:20:42.2186092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2186218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2186500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2186635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2186820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcz0j6a7
2022-11-23T02:20:42.2187069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcz0j6a7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2187325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2187330Z 
2022-11-23T02:20:42.2187401Z Running tests...
2022-11-23T02:20:42.2187594Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2187810Z   test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19300
2022-11-23T02:20:42.2187965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19301
2022-11-23T02:20:42.2188146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19302
2022-11-23T02:20:42.2188298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19303
2022-11-23T02:20:42.2188562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2188683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2189066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2189228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2189412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8hmac3l
2022-11-23T02:20:42.2189677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8hmac3l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2190075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2190198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2190480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2190604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2190787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2r1y4a4p
2022-11-23T02:20:42.2190978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2r1y4a4p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2191246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2191368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2191646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2191780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2191962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kregb3f
2022-11-23T02:20:42.2192143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kregb3f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2192410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2192530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2192807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2192940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2193123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokdrnpkr
2022-11-23T02:20:42.2193320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokdrnpkr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2193482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2193658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2193848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2194009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2194183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2194356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2194528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2194757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2195056Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2195355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2195645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2195920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2195987Z ok (2.675s)
2022-11-23T02:20:42.2195995Z 
2022-11-23T02:20:42.2196192Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2196264Z Ran 1 test in 2.675s
2022-11-23T02:20:42.2196270Z 
2022-11-23T02:20:42.2196330Z OK
2022-11-23T02:20:42.2196336Z 
2022-11-23T02:20:42.2196417Z Generating XML reports...
2022-11-23T02:20:42.2196757Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020345.xml
2022-11-23T02:20:42.2197023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2202708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2203058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2203201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2203393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphau5ex1d
2022-11-23T02:20:42.2203596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphau5ex1d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2203846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2203853Z 
2022-11-23T02:20:42.2203926Z Running tests...
2022-11-23T02:20:42.2204121Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2204334Z   test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19535
2022-11-23T02:20:42.2204485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19536
2022-11-23T02:20:42.2204640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19537
2022-11-23T02:20:42.2204795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19538
2022-11-23T02:20:42.2205169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2205280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2205570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2205706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2205894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6roj00e
2022-11-23T02:20:42.2206168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6roj00e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2206444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2206569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2206852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2206987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2207158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl_rsmf0
2022-11-23T02:20:42.2207409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl_rsmf0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2207762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2207951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2208259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2208396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2208639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp991val0n
2022-11-23T02:20:42.2208892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp991val0n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2209171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2209284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2209565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2209702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2209889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h_nou_s
2022-11-23T02:20:42.2210084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h_nou_s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2210247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2210410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2210572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2210721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2210899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2211076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2211252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2211425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2211726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2212025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2212319Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2212612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2212668Z ok (2.776s)
2022-11-23T02:20:42.2212686Z 
2022-11-23T02:20:42.2212870Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2212989Z Ran 1 test in 2.776s
2022-11-23T02:20:42.2212995Z 
2022-11-23T02:20:42.2213056Z OK
2022-11-23T02:20:42.2213061Z 
2022-11-23T02:20:42.2213145Z Generating XML reports...
2022-11-23T02:20:42.2213490Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020349.xml
2022-11-23T02:20:42.2213767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2213890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2214173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2214324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2214512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyklrlxyl
2022-11-23T02:20:42.2214716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyklrlxyl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2215132Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2215138Z 
2022-11-23T02:20:42.2215212Z Running tests...
2022-11-23T02:20:42.2215413Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2215626Z   test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19934
2022-11-23T02:20:42.2215783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19935
2022-11-23T02:20:42.2215926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19936
2022-11-23T02:20:42.2216081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19937
2022-11-23T02:20:42.2216359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2216483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2216764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2216899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2217084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xf8y1dh
2022-11-23T02:20:42.2217279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xf8y1dh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2217552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2217662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2217943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2218077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2218261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4v3r8ce
2022-11-23T02:20:42.2218454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4v3r8ce/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2218723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2218845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2219124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2219247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2219432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsqa7bbw
2022-11-23T02:20:42.2219625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsqa7bbw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2219964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2220088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2220369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2220502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2220687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3iylgy2
2022-11-23T02:20:42.2220920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3iylgy2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2221071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2221235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2221399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2221559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2221626Z ok (2.576s)
2022-11-23T02:20:42.2221631Z 
2022-11-23T02:20:42.2221831Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2221905Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2221910Z 
2022-11-23T02:20:42.2221969Z OK
2022-11-23T02:20:42.2221974Z 
2022-11-23T02:20:42.2222044Z Generating XML reports...
2022-11-23T02:20:42.2222390Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020354.xml
2022-11-23T02:20:42.2222665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2222788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2223068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2223205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2223389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99k0td9d
2022-11-23T02:20:42.2223582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99k0td9d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2223835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2223840Z 
2022-11-23T02:20:42.2223899Z Running tests...
2022-11-23T02:20:42.2224096Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2224311Z   test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20157
2022-11-23T02:20:42.2224467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20158
2022-11-23T02:20:42.2224622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20159
2022-11-23T02:20:42.2224775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20160
2022-11-23T02:20:42.2225048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2225171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2225451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2225572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2225758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlit0jba
2022-11-23T02:20:42.2225953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlit0jba/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2226255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2226381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2226661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2226796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2226977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ojj88l6
2022-11-23T02:20:42.2227156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ojj88l6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2227460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2227582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2227860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2227998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2228183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlw6jlis
2022-11-23T02:20:42.2228380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlw6jlis/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2228651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2228773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2229041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2229175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2229358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwh7o_m0
2022-11-23T02:20:42.2229553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwh7o_m0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2229718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2229879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2230040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2230201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2230255Z ok (2.172s)
2022-11-23T02:20:42.2230262Z 
2022-11-23T02:20:42.2230455Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2230528Z Ran 1 test in 2.172s
2022-11-23T02:20:42.2230533Z 
2022-11-23T02:20:42.2230591Z OK
2022-11-23T02:20:42.2230596Z 
2022-11-23T02:20:42.2230678Z Generating XML reports...
2022-11-23T02:20:42.2231020Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020358.xml
2022-11-23T02:20:42.2231293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2231418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2231696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2231816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2231995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49im1cg1
2022-11-23T02:20:42.2232188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49im1cg1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2232442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2232447Z 
2022-11-23T02:20:42.2232521Z Running tests...
2022-11-23T02:20:42.2232744Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2232967Z   test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20216
2022-11-23T02:20:42.2233122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20217
2022-11-23T02:20:42.2233263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20218
2022-11-23T02:20:42.2233418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20219
2022-11-23T02:20:42.2233692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2233847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2234128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2234264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2234449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gqqffxy
2022-11-23T02:20:42.2234643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gqqffxy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2234914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2235023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2235302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2235437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2235620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqt9vrznb
2022-11-23T02:20:42.2235814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqt9vrznb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2236085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2236207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2236484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2236604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2236786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuuaavfg
2022-11-23T02:20:42.2236981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuuaavfg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2237248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2237373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2237650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2237786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2237968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmn0wji9
2022-11-23T02:20:42.2238160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmn0wji9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2238311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2238473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2238637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2238797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2238863Z ok (2.472s)
2022-11-23T02:20:42.2238868Z 
2022-11-23T02:20:42.2239092Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2239166Z Ran 1 test in 2.472s
2022-11-23T02:20:42.2239171Z 
2022-11-23T02:20:42.2239232Z OK
2022-11-23T02:20:42.2239237Z 
2022-11-23T02:20:42.2239306Z Generating XML reports...
2022-11-23T02:20:42.2239644Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020402.xml
2022-11-23T02:20:42.2239916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2240037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2240345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2240478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2240660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94uim3zs
2022-11-23T02:20:42.2240854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94uim3zs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2241106Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2241111Z 
2022-11-23T02:20:42.2241170Z Running tests...
2022-11-23T02:20:42.2241364Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2241581Z   test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20439
2022-11-23T02:20:42.2241739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20440
2022-11-23T02:20:42.2241894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20441
2022-11-23T02:20:42.2242046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20442
2022-11-23T02:20:42.2242320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2242440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2242710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2242846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2243030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumknwo4h
2022-11-23T02:20:42.2243226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumknwo4h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2243502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2243624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2243908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2244045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2244231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvb53mp6
2022-11-23T02:20:42.2244414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvb53mp6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2244685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2244807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2245175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2245310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2245494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqlzr9n1
2022-11-23T02:20:42.2245724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqlzr9n1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2245995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2246117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2246383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2246518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2246700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xfo96vs
2022-11-23T02:20:42.2246935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xfo96vs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2247099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2247262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2247427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2247587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2247844Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2248114Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2248382Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2248645Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2248902Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2249160Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2249229Z ok (2.473s)
2022-11-23T02:20:42.2249237Z 
2022-11-23T02:20:42.2249438Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2249513Z Ran 1 test in 2.473s
2022-11-23T02:20:42.2249518Z 
2022-11-23T02:20:42.2249578Z OK
2022-11-23T02:20:42.2249583Z 
2022-11-23T02:20:42.2249665Z Generating XML reports...
2022-11-23T02:20:42.2249995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020407.xml
2022-11-23T02:20:42.2250269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2250391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2250675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2250811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2251000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxa6snvb_
2022-11-23T02:20:42.2251196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxa6snvb_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2251455Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2251460Z 
2022-11-23T02:20:42.2251531Z Running tests...
2022-11-23T02:20:42.2251742Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2251968Z   test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20662
2022-11-23T02:20:42.2252126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20663
2022-11-23T02:20:42.2252282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20664
2022-11-23T02:20:42.2252434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20665
2022-11-23T02:20:42.2252736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2252858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2253138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2253262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2253448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkntpfax
2022-11-23T02:20:42.2253644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkntpfax/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2253919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2254042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2254322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2254458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2254640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp398h7m6m
2022-11-23T02:20:42.2254834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp398h7m6m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2255191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2255313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2255591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2255725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2255909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxan0gjx1
2022-11-23T02:20:42.2256102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxan0gjx1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2256375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2256496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2256765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2256899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2257080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jgv6qfi
2022-11-23T02:20:42.2257272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jgv6qfi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2257434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2257599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2257761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2257920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2258146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2258310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2258484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2258789Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2258960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2259298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2259595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2259888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2260153Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2260414Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2260673Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2260927Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2261191Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2261446Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2261701Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.2261952Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.2262021Z ok (2.574s)
2022-11-23T02:20:42.2262027Z 
2022-11-23T02:20:42.2262226Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2262300Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2262306Z 
2022-11-23T02:20:42.2262369Z OK
2022-11-23T02:20:42.2262375Z 
2022-11-23T02:20:42.2262459Z Generating XML reports...
2022-11-23T02:20:42.2262800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020411.xml
2022-11-23T02:20:42.2263063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2263187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2263470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2263610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2263794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmroe252g
2022-11-23T02:20:42.2263989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmroe252g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2264271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2264277Z 
2022-11-23T02:20:42.2264350Z Running tests...
2022-11-23T02:20:42.2264532Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2264757Z   test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20897
2022-11-23T02:20:42.2264913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20898
2022-11-23T02:20:42.2265068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20899
2022-11-23T02:20:42.2265253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20900
2022-11-23T02:20:42.2265530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2265652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2265938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2266072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2266245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj5b1zk5
2022-11-23T02:20:42.2266439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj5b1zk5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2266716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2266843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2267125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2267260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2267449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi1waxqk
2022-11-23T02:20:42.2267644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi1waxqk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2267902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2268023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2268305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2268440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2268624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbmq01v3
2022-11-23T02:20:42.2268818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbmq01v3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2269090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2269213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2269491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2269612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2269794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyou_s3sq
2022-11-23T02:20:42.2269985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyou_s3sq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2270150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2270313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2270474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2270663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2270732Z ok (2.581s)
2022-11-23T02:20:42.2270737Z 
2022-11-23T02:20:42.2270918Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2270994Z Ran 1 test in 2.581s
2022-11-23T02:20:42.2270999Z 
2022-11-23T02:20:42.2271058Z OK
2022-11-23T02:20:42.2271063Z 
2022-11-23T02:20:42.2271145Z Generating XML reports...
2022-11-23T02:20:42.2271486Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020416.xml
2022-11-23T02:20:42.2271788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2271912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2272193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2272330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2272503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxonixnhv
2022-11-23T02:20:42.2272699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxonixnhv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2272956Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2272961Z 
2022-11-23T02:20:42.2273032Z Running tests...
2022-11-23T02:20:42.2273225Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2273446Z   test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21120
2022-11-23T02:20:42.2273600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21121
2022-11-23T02:20:42.2273753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21122
2022-11-23T02:20:42.2273893Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21123
2022-11-23T02:20:42.2274166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2274287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2274567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2274701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2274887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z4dsi0a
2022-11-23T02:20:42.2275082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z4dsi0a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2275353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2275477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2275746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2275881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2276063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgvwggly
2022-11-23T02:20:42.2276262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgvwggly/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2276533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2276659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2276937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2277103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2277290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwqdvt8l
2022-11-23T02:20:42.2277473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwqdvt8l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2277743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2277866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2278145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2278309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2278493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjxmh7zk
2022-11-23T02:20:42.2278687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjxmh7zk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2278853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2279002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2279164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2279324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2279390Z ok (2.575s)
2022-11-23T02:20:42.2279395Z 
2022-11-23T02:20:42.2279590Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2279666Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2279671Z 
2022-11-23T02:20:42.2279730Z OK
2022-11-23T02:20:42.2279735Z 
2022-11-23T02:20:42.2279815Z Generating XML reports...
2022-11-23T02:20:42.2280155Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020420.xml
2022-11-23T02:20:42.2280417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2280540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2280823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2280958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2281145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoa6lop_i
2022-11-23T02:20:42.2281344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoa6lop_i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2281602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2281607Z 
2022-11-23T02:20:42.2281679Z Running tests...
2022-11-23T02:20:42.2281865Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2282095Z   test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21343
2022-11-23T02:20:42.2282251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21344
2022-11-23T02:20:42.2282404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21345
2022-11-23T02:20:42.2282557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21346
2022-11-23T02:20:42.2282830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2282954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2283236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2283369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2283571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf87xpml
2022-11-23T02:20:42.2283770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf87xpml/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2284041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2284164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2284444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2284610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2284792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzjh_h6l
2022-11-23T02:20:42.2285067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzjh_h6l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2285334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2285457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2285737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2285869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2286052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24rugsdw
2022-11-23T02:20:42.2286243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24rugsdw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2286518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2286640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2286920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2287041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2287224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgaohkphv
2022-11-23T02:20:42.2287418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgaohkphv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2287581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2287744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2287908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2288069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2288135Z ok (2.673s)
2022-11-23T02:20:42.2288140Z 
2022-11-23T02:20:42.2288324Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2288400Z Ran 1 test in 2.673s
2022-11-23T02:20:42.2288405Z 
2022-11-23T02:20:42.2288465Z OK
2022-11-23T02:20:42.2288470Z 
2022-11-23T02:20:42.2288552Z Generating XML reports...
2022-11-23T02:20:42.2288890Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020424.xml
2022-11-23T02:20:42.2289162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2289286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2289568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2289701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2289872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rfk359d
2022-11-23T02:20:42.2290095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rfk359d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2290350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2290356Z 
2022-11-23T02:20:42.2290432Z Running tests...
2022-11-23T02:20:42.2290627Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2290861Z   test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21566
2022-11-23T02:20:42.2291017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21567
2022-11-23T02:20:42.2291245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21568
2022-11-23T02:20:42.2291398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21569
2022-11-23T02:20:42.2291659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2291782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2292065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2292199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2292383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5_28pgz
2022-11-23T02:20:42.2292575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5_28pgz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2292849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2292970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2293236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2293372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2293553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh3fko_x
2022-11-23T02:20:42.2293749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh3fko_x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2294022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2294144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2294423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2294558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2294742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_otil0tl
2022-11-23T02:20:42.2295124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_otil0tl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2295398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2295519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2295799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2295932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2296114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwcqsmv0
2022-11-23T02:20:42.2296308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwcqsmv0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2296471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2296620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2296833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2296996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2297063Z ok (2.577s)
2022-11-23T02:20:42.2297068Z 
2022-11-23T02:20:42.2297267Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2297343Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2297348Z 
2022-11-23T02:20:42.2297408Z OK
2022-11-23T02:20:42.2297413Z 
2022-11-23T02:20:42.2297494Z Generating XML reports...
2022-11-23T02:20:42.2297870Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020429.xml
2022-11-23T02:20:42.2298132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2298255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2298535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2298671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2298853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspww7eea
2022-11-23T02:20:42.2299051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspww7eea/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2299306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2299313Z 
2022-11-23T02:20:42.2299386Z Running tests...
2022-11-23T02:20:42.2299567Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2299790Z   test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21789
2022-11-23T02:20:42.2299947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21790
2022-11-23T02:20:42.2300102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21791
2022-11-23T02:20:42.2300253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21792
2022-11-23T02:20:42.2300527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2300650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2300927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2301064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2301235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi5mfqoa
2022-11-23T02:20:42.2301429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi5mfqoa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2301701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2301823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2302103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2302237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2302420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0a0kac7
2022-11-23T02:20:42.2302611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0a0kac7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2302870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2302991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2303295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2303430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2303613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptema6l49
2022-11-23T02:20:42.2303804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptema6l49/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2304075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2304194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2304498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2304618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2304799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70crczs9
2022-11-23T02:20:42.2304990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70crczs9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2305152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2305314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2305476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2305637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2305704Z ok (2.575s)
2022-11-23T02:20:42.2305710Z 
2022-11-23T02:20:42.2305892Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2305966Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2305972Z 
2022-11-23T02:20:42.2306031Z OK
2022-11-23T02:20:42.2306035Z 
2022-11-23T02:20:42.2306118Z Generating XML reports...
2022-11-23T02:20:42.2306458Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020433.xml
2022-11-23T02:20:42.2306728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2306850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2307129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2307262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2307435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpz_m9bx
2022-11-23T02:20:42.2307628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpz_m9bx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2307882Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2307887Z 
2022-11-23T02:20:42.2307961Z Running tests...
2022-11-23T02:20:42.2308158Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2308370Z   test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22012
2022-11-23T02:20:42.2308525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22013
2022-11-23T02:20:42.2308679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22014
2022-11-23T02:20:42.2308818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22015
2022-11-23T02:20:42.2309093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2309213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2309490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2309654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2309840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0_m3gbp
2022-11-23T02:20:42.2310035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0_m3gbp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2310304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2310428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2310695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2310857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2311040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsz56wnd1
2022-11-23T02:20:42.2311233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsz56wnd1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2311507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2311630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2311909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2312041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2312223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply3cyt7x
2022-11-23T02:20:42.2312404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply3cyt7x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2312674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2312794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2313075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2313208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2313389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfc92mlb8
2022-11-23T02:20:42.2313579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfc92mlb8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2313740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2313891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2314049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2314209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2314275Z ok (2.574s)
2022-11-23T02:20:42.2314282Z 
2022-11-23T02:20:42.2314479Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2314553Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2314558Z 
2022-11-23T02:20:42.2314617Z OK
2022-11-23T02:20:42.2314622Z 
2022-11-23T02:20:42.2314712Z Generating XML reports...
2022-11-23T02:20:42.2315036Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020438.xml
2022-11-23T02:20:42.2315308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2315436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2315712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2315845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2316054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kc34bv1
2022-11-23T02:20:42.2316247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kc34bv1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2316500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2316505Z 
2022-11-23T02:20:42.2316577Z Running tests...
2022-11-23T02:20:42.2316758Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2316982Z   test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22235
2022-11-23T02:20:42.2317167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22236
2022-11-23T02:20:42.2317323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22237
2022-11-23T02:20:42.2317476Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22238
2022-11-23T02:20:42.2317750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2317871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2318153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2318286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2318456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ir6sk3p
2022-11-23T02:20:42.2318651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ir6sk3p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2318921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2319042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2319324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2319459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2319644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlz7rvur
2022-11-23T02:20:42.2319837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlz7rvur/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2320094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2320219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2320496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2320630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2320815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jhjyl08
2022-11-23T02:20:42.2321004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jhjyl08/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2321273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2321393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2321669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2321788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2321973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn_1mp1o
2022-11-23T02:20:42.2322163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn_1mp1o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2322326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2322523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2322683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2322843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2322910Z ok (2.680s)
2022-11-23T02:20:42.2322916Z 
2022-11-23T02:20:42.2323100Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2323173Z Ran 1 test in 2.680s
2022-11-23T02:20:42.2323178Z 
2022-11-23T02:20:42.2323267Z OK
2022-11-23T02:20:42.2323272Z 
2022-11-23T02:20:42.2323354Z Generating XML reports...
2022-11-23T02:20:42.2323692Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020442.xml
2022-11-23T02:20:42.2323964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2324088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2324365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2324499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2324669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmiwrsumj
2022-11-23T02:20:42.2324866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmiwrsumj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2325212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2325217Z 
2022-11-23T02:20:42.2325294Z Running tests...
2022-11-23T02:20:42.2325489Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2325710Z   test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22494
2022-11-23T02:20:42.2325866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22495
2022-11-23T02:20:42.2326020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22496
2022-11-23T02:20:42.2326159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22497
2022-11-23T02:20:42.2326432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2326555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2326838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2326972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2327158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppybbm3kz
2022-11-23T02:20:42.2327356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppybbm3kz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2327630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2327752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2328018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2328153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2328337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uu9ggfv
2022-11-23T02:20:42.2328533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uu9ggfv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2328804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2328958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2329237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2329370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2329555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeeabbrbo
2022-11-23T02:20:42.2329813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2329935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2330240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2330375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2330559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5ju62yl
2022-11-23T02:20:42.2330755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeabbrbo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2330944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5ju62yl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2331108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2331257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2331419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2331581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2331647Z ok (2.479s)
2022-11-23T02:20:42.2331651Z 
2022-11-23T02:20:42.2331846Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2331921Z Ran 1 test in 2.479s
2022-11-23T02:20:42.2331926Z 
2022-11-23T02:20:42.2331985Z OK
2022-11-23T02:20:42.2331990Z 
2022-11-23T02:20:42.2332074Z Generating XML reports...
2022-11-23T02:20:42.2332403Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020447.xml
2022-11-23T02:20:42.2332675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2332798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2333074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2333212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2333397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpata_mrn1
2022-11-23T02:20:42.2333593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpata_mrn1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2333848Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2333853Z 
2022-11-23T02:20:42.2333924Z Running tests...
2022-11-23T02:20:42.2334106Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2334319Z   test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22717
2022-11-23T02:20:42.2334473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22718
2022-11-23T02:20:42.2334625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22719
2022-11-23T02:20:42.2334779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22720
2022-11-23T02:20:42.2335172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2335295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2335623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2335761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2335934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpsj_t6m
2022-11-23T02:20:42.2336129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpsj_t6m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2336402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2336573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2336853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2336986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2337172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ect5bz6
2022-11-23T02:20:42.2337365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ect5bz6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2337624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2337747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2338026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2338160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2338345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbdkdmjr
2022-11-23T02:20:42.2338541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbdkdmjr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2338811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2338934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2339215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2339335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2339517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d12uhd0
2022-11-23T02:20:42.2339711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d12uhd0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2339875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2340037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2340200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2340360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2340426Z ok (2.575s)
2022-11-23T02:20:42.2340431Z 
2022-11-23T02:20:42.2340616Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2340692Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2340697Z 
2022-11-23T02:20:42.2340756Z OK
2022-11-23T02:20:42.2340761Z 
2022-11-23T02:20:42.2340843Z Generating XML reports...
2022-11-23T02:20:42.2341180Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020451.xml
2022-11-23T02:20:42.2341456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2341577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2341857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2342027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2342201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpul3b9pgw
2022-11-23T02:20:42.2342396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpul3b9pgw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2342649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2342654Z 
2022-11-23T02:20:42.2342726Z Running tests...
2022-11-23T02:20:42.2342922Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2343167Z   test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22940
2022-11-23T02:20:42.2343323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22941
2022-11-23T02:20:42.2343475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22942
2022-11-23T02:20:42.2343616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22943
2022-11-23T02:20:42.2343891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2344013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2344294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2344428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2344614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30txpo5a
2022-11-23T02:20:42.2344812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30txpo5a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2345142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2345266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2345534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2345669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2345854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuluxcd3
2022-11-23T02:20:42.2346047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuluxcd3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2346318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2346441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2346718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2346850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2347021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt_feuee
2022-11-23T02:20:42.2347214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt_feuee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2347483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2347605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2347886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2348023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2348206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08reu2qh
2022-11-23T02:20:42.2348397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08reu2qh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2348601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2348753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2348911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2349069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2349138Z ok (2.574s)
2022-11-23T02:20:42.2349143Z 
2022-11-23T02:20:42.2349340Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2349445Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2349450Z 
2022-11-23T02:20:42.2349509Z OK
2022-11-23T02:20:42.2349513Z 
2022-11-23T02:20:42.2349596Z Generating XML reports...
2022-11-23T02:20:42.2349921Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020456.xml
2022-11-23T02:20:42.2350195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2350317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2350600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2350733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2350917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg29unavl
2022-11-23T02:20:42.2351112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg29unavl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2351370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2351376Z 
2022-11-23T02:20:42.2351447Z Running tests...
2022-11-23T02:20:42.2351630Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2351855Z   test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23163
2022-11-23T02:20:42.2352009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23164
2022-11-23T02:20:42.2352163Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23165
2022-11-23T02:20:42.2352314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23166
2022-11-23T02:20:42.2352590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2352714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2352995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2353128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2353302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvwoj08j
2022-11-23T02:20:42.2353496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvwoj08j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2353765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2353890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2354170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2354304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2354490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_w7tjcq
2022-11-23T02:20:42.2354683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_w7tjcq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2354970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2355094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2355371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2355505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2355692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyt7p39y
2022-11-23T02:20:42.2355884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyt7p39y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2356181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2356302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2356579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2356702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2356887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2pztr7a
2022-11-23T02:20:42.2357083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2pztr7a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2357247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2357408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2357570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2357731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2357796Z ok (2.675s)
2022-11-23T02:20:42.2357801Z 
2022-11-23T02:20:42.2357983Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2358058Z Ran 1 test in 2.675s
2022-11-23T02:20:42.2358063Z 
2022-11-23T02:20:42.2358125Z OK
2022-11-23T02:20:42.2358129Z 
2022-11-23T02:20:42.2358212Z Generating XML reports...
2022-11-23T02:20:42.2358550Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020500.xml
2022-11-23T02:20:42.2358821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2358943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2359222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2359357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2359529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgglj3g3
2022-11-23T02:20:42.2359727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgglj3g3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2359982Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2359987Z 
2022-11-23T02:20:42.2360058Z Running tests...
2022-11-23T02:20:42.2360255Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2360474Z   test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23386
2022-11-23T02:20:42.2360630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23387
2022-11-23T02:20:42.2360786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23388
2022-11-23T02:20:42.2360925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23389
2022-11-23T02:20:42.2361202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2361354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2361639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2361775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2361960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_usa1y0
2022-11-23T02:20:42.2362153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_usa1y0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2362423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2362573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2362839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2362973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2363158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ppgsyhy
2022-11-23T02:20:42.2363352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ppgsyhy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2363624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2363745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2364023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2364158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2364328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_9_np2q
2022-11-23T02:20:42.2364520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_9_np2q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2364790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2364977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2365262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2365394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2365575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yc2uwrz
2022-11-23T02:20:42.2365769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yc2uwrz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2365931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2366081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2366244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2366404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2366470Z ok (2.575s)
2022-11-23T02:20:42.2366475Z 
2022-11-23T02:20:42.2366673Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2366749Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2366754Z 
2022-11-23T02:20:42.2366813Z OK
2022-11-23T02:20:42.2366818Z 
2022-11-23T02:20:42.2366901Z Generating XML reports...
2022-11-23T02:20:42.2367228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020505.xml
2022-11-23T02:20:42.2367501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2367623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2367937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2368072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2368255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp147__8k8
2022-11-23T02:20:42.2368443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp147__8k8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2368697Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2368702Z 
2022-11-23T02:20:42.2368775Z Running tests...
2022-11-23T02:20:42.2368985Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2369259Z   test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23609
2022-11-23T02:20:42.2369417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23610
2022-11-23T02:20:42.2369573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23611
2022-11-23T02:20:42.2369724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23612
2022-11-23T02:20:42.2369999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2370121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2370403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2370540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2370710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kqq7r2o
2022-11-23T02:20:42.2370904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kqq7r2o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2371181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2371304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2371585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2371719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2371903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3weudrs
2022-11-23T02:20:42.2372101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3weudrs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2372358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2372482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2372763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2372896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2373078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk_4eyp4
2022-11-23T02:20:42.2373271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk_4eyp4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2373541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2373663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2373946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2374067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2374249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qn42z02
2022-11-23T02:20:42.2374468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qn42z02/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2374632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2374797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2375061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2375223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2375338Z ok (2.579s)
2022-11-23T02:20:42.2375343Z 
2022-11-23T02:20:42.2375530Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2375605Z Ran 1 test in 2.579s
2022-11-23T02:20:42.2375610Z 
2022-11-23T02:20:42.2375669Z OK
2022-11-23T02:20:42.2375674Z 
2022-11-23T02:20:42.2375758Z Generating XML reports...
2022-11-23T02:20:42.2376102Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020509.xml
2022-11-23T02:20:42.2376375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2376498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2376781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2376985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2377187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4h25awc
2022-11-23T02:20:42.2377381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4h25awc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2377637Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2377642Z 
2022-11-23T02:20:42.2377716Z Running tests...
2022-11-23T02:20:42.2377912Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2378124Z   test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23832
2022-11-23T02:20:42.2378278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23833
2022-11-23T02:20:42.2378431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23834
2022-11-23T02:20:42.2378570Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23835
2022-11-23T02:20:42.2378845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2378967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2379277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2379471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2379660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zcki1km
2022-11-23T02:20:42.2379935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2380057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2380339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2380462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2380647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmj8ezx_
2022-11-23T02:20:42.2380843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zcki1km/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2381084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmj8ezx_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2381357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2381478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2381757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2381889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2382058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqruetkig
2022-11-23T02:20:42.2382301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqruetkig/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2382571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2382693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2382974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2383106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2383288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ayz2srt
2022-11-23T02:20:42.2383478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ayz2srt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2383643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2383797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2383960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2384121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2384187Z ok (2.475s)
2022-11-23T02:20:42.2384194Z 
2022-11-23T02:20:42.2384393Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2384467Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2384473Z 
2022-11-23T02:20:42.2384532Z OK
2022-11-23T02:20:42.2384537Z 
2022-11-23T02:20:42.2384618Z Generating XML reports...
2022-11-23T02:20:42.2384944Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020513.xml
2022-11-23T02:20:42.2385217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2385341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2385620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2385753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2385940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrea5ezz
2022-11-23T02:20:42.2386139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrea5ezz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2386394Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2386399Z 
2022-11-23T02:20:42.2386470Z Running tests...
2022-11-23T02:20:42.2386650Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2386869Z   test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24055
2022-11-23T02:20:42.2387027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24056
2022-11-23T02:20:42.2387180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24057
2022-11-23T02:20:42.2387332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24058
2022-11-23T02:20:42.2387653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2387777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2388060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2388194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2388366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwv4wu90
2022-11-23T02:20:42.2388561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwv4wu90/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2388862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2388987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2389271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2389405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2389589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ermuxfc
2022-11-23T02:20:42.2389784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ermuxfc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2390042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2390166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2390451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2390586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2390769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld8b1e7g
2022-11-23T02:20:42.2390967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld8b1e7g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2391241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2391365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2391648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2391768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2391952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidga7l3f
2022-11-23T02:20:42.2392146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidga7l3f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2392311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2392476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2392638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2392798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2392864Z ok (2.577s)
2022-11-23T02:20:42.2392869Z 
2022-11-23T02:20:42.2393052Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2393127Z Ran 1 test in 2.578s
2022-11-23T02:20:42.2393132Z 
2022-11-23T02:20:42.2393191Z OK
2022-11-23T02:20:42.2393198Z 
2022-11-23T02:20:42.2393281Z Generating XML reports...
2022-11-23T02:20:42.2393620Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020518.xml
2022-11-23T02:20:42.2393893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2394077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2394360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2394494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2394666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lrxyom5
2022-11-23T02:20:42.2394863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lrxyom5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2395118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2395150Z 
2022-11-23T02:20:42.2395224Z Running tests...
2022-11-23T02:20:42.2395424Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2395652Z   test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24278
2022-11-23T02:20:42.2395806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24279
2022-11-23T02:20:42.2395959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24280
2022-11-23T02:20:42.2396105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24281
2022-11-23T02:20:42.2396380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2396500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2396781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2396918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2397102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdvdkp5l
2022-11-23T02:20:42.2397299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdvdkp5l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2397573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2397695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2397960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2398094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2398278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpbdymfy
2022-11-23T02:20:42.2398475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpbdymfy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2398748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2398872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2399151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2399286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2399456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mz3zrqt
2022-11-23T02:20:42.2399649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mz3zrqt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2399920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2400044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2400325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2400458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2400674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp4ltj4w
2022-11-23T02:20:42.2400869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp4ltj4w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2401032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2401182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2401343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2401537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2401602Z ok (2.677s)
2022-11-23T02:20:42.2401607Z 
2022-11-23T02:20:42.2401801Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2401875Z Ran 1 test in 2.677s
2022-11-23T02:20:42.2401880Z 
2022-11-23T02:20:42.2401940Z OK
2022-11-23T02:20:42.2401945Z 
2022-11-23T02:20:42.2402030Z Generating XML reports...
2022-11-23T02:20:42.2402356Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020522.xml
2022-11-23T02:20:42.2402631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2402752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2403032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2403169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2403352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfb82ozj
2022-11-23T02:20:42.2403548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfb82ozj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2403803Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2403808Z 
2022-11-23T02:20:42.2403881Z Running tests...
2022-11-23T02:20:42.2404061Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2404280Z   test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24521
2022-11-23T02:20:42.2404437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24522
2022-11-23T02:20:42.2404590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24523
2022-11-23T02:20:42.2404745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24524
2022-11-23T02:20:42.2405095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2405221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2405506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2405641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2405812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61hm5f5k
2022-11-23T02:20:42.2406005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61hm5f5k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2406281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2406405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2406686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2406820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2407041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeth6g8pt
2022-11-23T02:20:42.2407238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeth6g8pt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2407499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2407622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2407900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2408034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2408245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_ckxzhe
2022-11-23T02:20:42.2408436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_ckxzhe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2408708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2408830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2409110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2409231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2409414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5ak7_8z
2022-11-23T02:20:42.2409605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5ak7_8z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2409773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2409936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2410096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2410257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2410323Z ok (2.578s)
2022-11-23T02:20:42.2410328Z 
2022-11-23T02:20:42.2410510Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2410586Z Ran 1 test in 2.578s
2022-11-23T02:20:42.2410590Z 
2022-11-23T02:20:42.2410650Z OK
2022-11-23T02:20:42.2410655Z 
2022-11-23T02:20:42.2410737Z Generating XML reports...
2022-11-23T02:20:42.2411076Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020527.xml
2022-11-23T02:20:42.2411347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2411469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2411751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2411886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2412057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa16kbc40
2022-11-23T02:20:42.2412250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa16kbc40/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2412503Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2412509Z 
2022-11-23T02:20:42.2412582Z Running tests...
2022-11-23T02:20:42.2412778Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2413000Z   test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24744
2022-11-23T02:20:42.2413155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24745
2022-11-23T02:20:42.2413309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24746
2022-11-23T02:20:42.2413478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24747
2022-11-23T02:20:42.2413753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2413874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2414154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2414288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2414473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01d8qd31
2022-11-23T02:20:42.2414696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01d8qd31/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2415106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2415232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2415500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2415635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2415818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqn11vnf
2022-11-23T02:20:42.2416012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqn11vnf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2416280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2416404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2416680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2416813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2416984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot9fldq8
2022-11-23T02:20:42.2417176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot9fldq8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2417449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2417569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2417847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2417982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2418163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_hy718m
2022-11-23T02:20:42.2418352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_hy718m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2418518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2418669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2418828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2418991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2419168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2419343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2419518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2419693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2420039Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2420342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2420620Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2420911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2420978Z ok (2.274s)
2022-11-23T02:20:42.2421018Z 
2022-11-23T02:20:42.2421216Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2421290Z Ran 1 test in 2.275s
2022-11-23T02:20:42.2421295Z 
2022-11-23T02:20:42.2421355Z OK
2022-11-23T02:20:42.2421360Z 
2022-11-23T02:20:42.2421442Z Generating XML reports...
2022-11-23T02:20:42.2421786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020531.xml
2022-11-23T02:20:42.2422057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2422167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2422445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2422577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2422761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ff2kbnh
2022-11-23T02:20:42.2422958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ff2kbnh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2423210Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2423215Z 
2022-11-23T02:20:42.2423286Z Running tests...
2022-11-23T02:20:42.2423483Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2423687Z   test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24815
2022-11-23T02:20:42.2423842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24816
2022-11-23T02:20:42.2423996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24817
2022-11-23T02:20:42.2424146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24818
2022-11-23T02:20:42.2424422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2424548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2424829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2424964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2425153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj77ngxp3
2022-11-23T02:20:42.2425337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj77ngxp3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2425610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2425733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2426015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2426151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2426337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjfff6xp
2022-11-23T02:20:42.2426532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjfff6xp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2426839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2426952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2427235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2427370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2427552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgzyk88w
2022-11-23T02:20:42.2427794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgzyk88w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2428065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2428187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2428469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2428603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2428774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa342kcg8
2022-11-23T02:20:42.2428965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa342kcg8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2429128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2429291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2429456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2429618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2429685Z ok (2.480s)
2022-11-23T02:20:42.2429690Z 
2022-11-23T02:20:42.2429886Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2429947Z Ran 1 test in 2.480s
2022-11-23T02:20:42.2429966Z 
2022-11-23T02:20:42.2430012Z OK
2022-11-23T02:20:42.2430017Z 
2022-11-23T02:20:42.2430099Z Generating XML reports...
2022-11-23T02:20:42.2430441Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020535.xml
2022-11-23T02:20:42.2430714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2430837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2431121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2431255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2431440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xdeqqcy
2022-11-23T02:20:42.2431625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xdeqqcy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2431881Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2431886Z 
2022-11-23T02:20:42.2431966Z Running tests...
2022-11-23T02:20:42.2432161Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2432385Z   test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25038
2022-11-23T02:20:42.2432531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25039
2022-11-23T02:20:42.2432684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25040
2022-11-23T02:20:42.2432841Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25041
2022-11-23T02:20:42.2433149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2433276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2433557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2433690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2433873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jrafxqf
2022-11-23T02:20:42.2434055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jrafxqf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2434356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2434477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2434757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2434891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2435073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69yxgz60
2022-11-23T02:20:42.2435265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69yxgz60/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2435535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2435656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2435925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2436059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2436240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81v0r__e
2022-11-23T02:20:42.2436512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2436632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2436908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2437041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2437223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7r05wnz
2022-11-23T02:20:42.2437400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81v0r__e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2437591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7r05wnz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2437755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2437920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2438082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2438240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2438304Z ok (2.475s)
2022-11-23T02:20:42.2438309Z 
2022-11-23T02:20:42.2438505Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2438578Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2438583Z 
2022-11-23T02:20:42.2438630Z OK
2022-11-23T02:20:42.2438634Z 
2022-11-23T02:20:42.2438718Z Generating XML reports...
2022-11-23T02:20:42.2439056Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020540.xml
2022-11-23T02:20:42.2439328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2439482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2439759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2439890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2440071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouj61zo5
2022-11-23T02:20:42.2440251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouj61zo5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2440503Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2440535Z 
2022-11-23T02:20:42.2440607Z Running tests...
2022-11-23T02:20:42.2440802Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2441038Z   test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25261
2022-11-23T02:20:42.2441195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25262
2022-11-23T02:20:42.2441349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25263
2022-11-23T02:20:42.2441499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25264
2022-11-23T02:20:42.2441770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2441879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2442161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2442295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2442482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjas5olbx
2022-11-23T02:20:42.2442681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjas5olbx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2442951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2443073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2443350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2443471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2443653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vb_taac
2022-11-23T02:20:42.2443848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vb_taac/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2444120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2444243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2444526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2444658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2444841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d7tva47
2022-11-23T02:20:42.2445112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d7tva47/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2445375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2445499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2445777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2445912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2446131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1cmhmul
2022-11-23T02:20:42.2446326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1cmhmul/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2446486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2446649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2446807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2446955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2447050Z ok (2.578s)
2022-11-23T02:20:42.2447056Z 
2022-11-23T02:20:42.2447254Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2447331Z Ran 1 test in 2.578s
2022-11-23T02:20:42.2447336Z 
2022-11-23T02:20:42.2447395Z OK
2022-11-23T02:20:42.2447399Z 
2022-11-23T02:20:42.2447484Z Generating XML reports...
2022-11-23T02:20:42.2447823Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020544.xml
2022-11-23T02:20:42.2448092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2448202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2448480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2448613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2448806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl343ps_s
2022-11-23T02:20:42.2449001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl343ps_s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2449257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2449262Z 
2022-11-23T02:20:42.2449333Z Running tests...
2022-11-23T02:20:42.2449526Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2449781Z   test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25484
2022-11-23T02:20:42.2449925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25485
2022-11-23T02:20:42.2450077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25486
2022-11-23T02:20:42.2450233Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25487
2022-11-23T02:20:42.2450506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2450628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2450912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2451044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2451227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz117f_bl
2022-11-23T02:20:42.2451406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz117f_bl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2451677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2451799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2452081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2452212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2452425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvfrpkkf
2022-11-23T02:20:42.2452620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvfrpkkf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2452890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2453011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2453278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2453410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2453622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vvxsa0a
2022-11-23T02:20:42.2453816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vvxsa0a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2454087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2454209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2454486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2454617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2454787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qjw12h9
2022-11-23T02:20:42.2455091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qjw12h9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2455257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2455417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2455577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2455739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2455805Z ok (2.576s)
2022-11-23T02:20:42.2455810Z 
2022-11-23T02:20:42.2456004Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2456077Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2456083Z 
2022-11-23T02:20:42.2456130Z OK
2022-11-23T02:20:42.2456135Z 
2022-11-23T02:20:42.2456217Z Generating XML reports...
2022-11-23T02:20:42.2456555Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020549.xml
2022-11-23T02:20:42.2456830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2456950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2457228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2457370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2457553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4js3ebj
2022-11-23T02:20:42.2457733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4js3ebj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2457986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2457991Z 
2022-11-23T02:20:42.2458063Z Running tests...
2022-11-23T02:20:42.2458260Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2458490Z   test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25707
2022-11-23T02:20:42.2458645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25708
2022-11-23T02:20:42.2458798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25709
2022-11-23T02:20:42.2459006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25710
2022-11-23T02:20:42.2459281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2459390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2459670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2459804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2459989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27yscgvk
2022-11-23T02:20:42.2460219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27yscgvk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2460493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2460617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2460896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2461031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2461206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p0w_n3l
2022-11-23T02:20:42.2461396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p0w_n3l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2461666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2461789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2462069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2462204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2462389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxlvh_tx
2022-11-23T02:20:42.2462583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxlvh_tx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2462842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2462963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2463243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2463378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2463562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0x78khy
2022-11-23T02:20:42.2463755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0x78khy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2463919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2464079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2464240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2464389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2464454Z ok (2.482s)
2022-11-23T02:20:42.2464459Z 
2022-11-23T02:20:42.2464655Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2464733Z Ran 1 test in 2.483s
2022-11-23T02:20:42.2464738Z 
2022-11-23T02:20:42.2464796Z OK
2022-11-23T02:20:42.2464801Z 
2022-11-23T02:20:42.2464882Z Generating XML reports...
2022-11-23T02:20:42.2465221Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020553.xml
2022-11-23T02:20:42.2465531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2465642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2465919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2466054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2466236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce849nr7
2022-11-23T02:20:42.2466428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce849nr7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2466713Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2466719Z 
2022-11-23T02:20:42.2466790Z Running tests...
2022-11-23T02:20:42.2466983Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2467208Z   test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25936
2022-11-23T02:20:42.2467350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25937
2022-11-23T02:20:42.2467504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25938
2022-11-23T02:20:42.2467657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25939
2022-11-23T02:20:42.2467929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2468053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2468332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2468465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2468655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4gapxv5
2022-11-23T02:20:42.2468839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4gapxv5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2469108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2469230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2469511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2469645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2469831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsok1vx6j
2022-11-23T02:20:42.2470102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2470221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2470501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2470621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2470802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7o5y955m
2022-11-23T02:20:42.2470997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsok1vx6j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2471183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7o5y955m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2471456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2471576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2471854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2472019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2472192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad79rl04
2022-11-23T02:20:42.2472383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad79rl04/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2472545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2472708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2472913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2473075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2473140Z ok (2.475s)
2022-11-23T02:20:42.2473145Z 
2022-11-23T02:20:42.2473339Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2473414Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2473420Z 
2022-11-23T02:20:42.2473467Z OK
2022-11-23T02:20:42.2473471Z 
2022-11-23T02:20:42.2473552Z Generating XML reports...
2022-11-23T02:20:42.2473886Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020557.xml
2022-11-23T02:20:42.2474157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2474279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2474557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2474691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2474875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j2uism0
2022-11-23T02:20:42.2475059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j2uism0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2475313Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2475318Z 
2022-11-23T02:20:42.2475388Z Running tests...
2022-11-23T02:20:42.2475581Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2475814Z   test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26159
2022-11-23T02:20:42.2475966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26160
2022-11-23T02:20:42.2476121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26161
2022-11-23T02:20:42.2476271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26162
2022-11-23T02:20:42.2476545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2476655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2476935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2477071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2477253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsjo_nmw
2022-11-23T02:20:42.2477449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsjo_nmw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2477831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2477958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2478240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2478416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2478586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jmjbz49
2022-11-23T02:20:42.2478777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jmjbz49/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2479047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2479169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2479448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2479613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2479796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqtmb_ip
2022-11-23T02:20:42.2479988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqtmb_ip/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2480249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2480370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2480646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2480781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2480962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm6_hhd5
2022-11-23T02:20:42.2481158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm6_hhd5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2481321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2481485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2481647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2481795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2481860Z ok (2.477s)
2022-11-23T02:20:42.2481865Z 
2022-11-23T02:20:42.2482059Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2482136Z Ran 1 test in 2.477s
2022-11-23T02:20:42.2482140Z 
2022-11-23T02:20:42.2482200Z OK
2022-11-23T02:20:42.2482204Z 
2022-11-23T02:20:42.2482285Z Generating XML reports...
2022-11-23T02:20:42.2482626Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020602.xml
2022-11-23T02:20:42.2482897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2483007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2483287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2483422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2483606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw2554d5
2022-11-23T02:20:42.2483801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw2554d5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2484053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2484058Z 
2022-11-23T02:20:42.2484132Z Running tests...
2022-11-23T02:20:42.2484326Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2484560Z   test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26382
2022-11-23T02:20:42.2484703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26383
2022-11-23T02:20:42.2484886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26384
2022-11-23T02:20:42.2485103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26385
2022-11-23T02:20:42.2485381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2485505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2485786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2485950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2486135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae_6n1sf
2022-11-23T02:20:42.2486314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae_6n1sf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2486586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2486708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2486989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2487122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2487305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozrla28n
2022-11-23T02:20:42.2487499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozrla28n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2487770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2487891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2488158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2488290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2488472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3lbnogn
2022-11-23T02:20:42.2488663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3lbnogn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2488933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2489054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2489335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2489467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2489637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8q3kjogh
2022-11-23T02:20:42.2489832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8q3kjogh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2489994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2490160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2490318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2490477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2490547Z ok (2.473s)
2022-11-23T02:20:42.2490552Z 
2022-11-23T02:20:42.2490746Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2490819Z Ran 1 test in 2.473s
2022-11-23T02:20:42.2490825Z 
2022-11-23T02:20:42.2490871Z OK
2022-11-23T02:20:42.2490875Z 
2022-11-23T02:20:42.2490960Z Generating XML reports...
2022-11-23T02:20:42.2491327Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020606.xml
2022-11-23T02:20:42.2491601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2491724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2492001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2492135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2492381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6z_nzlj
2022-11-23T02:20:42.2492560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6z_nzlj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2492814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2492821Z 
2022-11-23T02:20:42.2492894Z Running tests...
2022-11-23T02:20:42.2493091Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2493340Z   test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26605
2022-11-23T02:20:42.2493496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26606
2022-11-23T02:20:42.2493651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26607
2022-11-23T02:20:42.2493803Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26608
2022-11-23T02:20:42.2494079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2494189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2494469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2494604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2494788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp572wws89
2022-11-23T02:20:42.2495092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp572wws89/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2495364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2495486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2495770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2495905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2496075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4ze5_ct
2022-11-23T02:20:42.2496271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4ze5_ct/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2496543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2496663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2496941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2497073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2497255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_gklw48
2022-11-23T02:20:42.2497448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_gklw48/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2497706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2497881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2498161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2498295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2498478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vy0alr1
2022-11-23T02:20:42.2498668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vy0alr1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2498830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2499026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2499187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2499335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2499402Z ok (2.572s)
2022-11-23T02:20:42.2499407Z 
2022-11-23T02:20:42.2499603Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2499678Z Ran 1 test in 2.572s
2022-11-23T02:20:42.2499683Z 
2022-11-23T02:20:42.2499741Z OK
2022-11-23T02:20:42.2499746Z 
2022-11-23T02:20:42.2499827Z Generating XML reports...
2022-11-23T02:20:42.2500166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020610.xml
2022-11-23T02:20:42.2500437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2500549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2500827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2500960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2501146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcktfyk8j
2022-11-23T02:20:42.2501343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcktfyk8j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2501593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2501599Z 
2022-11-23T02:20:42.2501671Z Running tests...
2022-11-23T02:20:42.2501864Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2502093Z   test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26828
2022-11-23T02:20:42.2502240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26829
2022-11-23T02:20:42.2502425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26830
2022-11-23T02:20:42.2502580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26831
2022-11-23T02:20:42.2502856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2502976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2503257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2503389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2503573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtrd3nre
2022-11-23T02:20:42.2503756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtrd3nre/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2504026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2504145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2504461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2504595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2504778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44qor4uu
2022-11-23T02:20:42.2504968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44qor4uu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2505238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2505385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2505652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2505785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2505969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85bojpo8
2022-11-23T02:20:42.2506159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85bojpo8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2506428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2506547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2506822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2506953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2507124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe_qj0k8
2022-11-23T02:20:42.2507312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe_qj0k8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2507472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2507636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2507797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2507956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2508207Z STAGE:2022-11-23 02:06:19 26829:26829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:20:42.2508428Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2022-11-23T02:20:42.2508681Z STAGE:2022-11-23 02:06:20 26829:26829 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:20:42.2508928Z STAGE:2022-11-23 02:06:20 26829:26829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:20:42.2508997Z ok (5.479s)
2022-11-23T02:20:42.2509001Z 
2022-11-23T02:20:42.2509197Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2509274Z Ran 1 test in 5.479s
2022-11-23T02:20:42.2509279Z 
2022-11-23T02:20:42.2509338Z OK
2022-11-23T02:20:42.2509342Z 
2022-11-23T02:20:42.2509425Z Generating XML reports...
2022-11-23T02:20:42.2509762Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020615.xml
2022-11-23T02:20:42.2510138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2510265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2510532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2510666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2510963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5phsds9z
2022-11-23T02:20:42.2511163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5phsds9z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2511417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2511422Z 
2022-11-23T02:20:42.2511492Z Running tests...
2022-11-23T02:20:42.2511686Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2511931Z   test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27052
2022-11-23T02:20:42.2512117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27053
2022-11-23T02:20:42.2512258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27054
2022-11-23T02:20:42.2512409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27055
2022-11-23T02:20:42.2512683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2512804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2513085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2513218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2513402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto71blld
2022-11-23T02:20:42.2513595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto71blld/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2513855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2513976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2514255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2514386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2514568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm_ye246
2022-11-23T02:20:42.2514758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm_ye246/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2515028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2515149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2515430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2515549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2515729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8_dj5x4
2022-11-23T02:20:42.2515920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8_dj5x4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2516193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2516316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2516591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2516723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2516904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdc140l9
2022-11-23T02:20:42.2517082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdc140l9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2517245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2517436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2517600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2517759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2518008Z STAGE:2022-11-23 02:06:26 27053:27053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:20:42.2518225Z [W utils.cpp:175] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2022-11-23T02:20:42.2518519Z STAGE:2022-11-23 02:06:27 27053:27053 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:20:42.2518775Z STAGE:2022-11-23 02:06:27 27053:27053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:20:42.2518829Z ok (5.479s)
2022-11-23T02:20:42.2518833Z 
2022-11-23T02:20:42.2519029Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2519102Z Ran 1 test in 5.479s
2022-11-23T02:20:42.2519107Z 
2022-11-23T02:20:42.2519167Z OK
2022-11-23T02:20:42.2519171Z 
2022-11-23T02:20:42.2519254Z Generating XML reports...
2022-11-23T02:20:42.2519589Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020622.xml
2022-11-23T02:20:42.2519860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2519980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2520260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2520384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2520566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtfxe6v8
2022-11-23T02:20:42.2520761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtfxe6v8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2521011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2521016Z 
2022-11-23T02:20:42.2521087Z Running tests...
2022-11-23T02:20:42.2521281Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2521512Z   test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27276
2022-11-23T02:20:42.2521668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27277
2022-11-23T02:20:42.2521808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27278
2022-11-23T02:20:42.2521960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27279
2022-11-23T02:20:42.2522236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2522357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2522638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2522773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2522958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8q5rqh4
2022-11-23T02:20:42.2523151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8q5rqh4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2523423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2523532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2523811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2523975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2524161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6merjj24
2022-11-23T02:20:42.2524354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6merjj24/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2524625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2524746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2525113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2525236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2525419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65y53xqy
2022-11-23T02:20:42.2525613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65y53xqy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2525886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2526009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2526288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2526420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2526603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu3d3j5q
2022-11-23T02:20:42.2526799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu3d3j5q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2526950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2527112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2527273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2527432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2527497Z ok (2.474s)
2022-11-23T02:20:42.2527502Z 
2022-11-23T02:20:42.2527699Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2527772Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2527778Z 
2022-11-23T02:20:42.2527837Z OK
2022-11-23T02:20:42.2527842Z 
2022-11-23T02:20:42.2527911Z Generating XML reports...
2022-11-23T02:20:42.2528253Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020630.xml
2022-11-23T02:20:42.2528528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2528653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2528933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2529066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2529251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6juaw9y
2022-11-23T02:20:42.2529445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6juaw9y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2529698Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2529705Z 
2022-11-23T02:20:42.2529763Z Running tests...
2022-11-23T02:20:42.2529960Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2530213Z   test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27499
2022-11-23T02:20:42.2530413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27500
2022-11-23T02:20:42.2530569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27501
2022-11-23T02:20:42.2530720Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27502
2022-11-23T02:20:42.2530993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2531114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2531396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2531547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2531731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeh15i8s_
2022-11-23T02:20:42.2531927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeh15i8s_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2532197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2532320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2532597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2532729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2532910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5vtxa4q
2022-11-23T02:20:42.2533091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5vtxa4q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2533360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2533481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2533763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2533896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2534080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvc3f02gi
2022-11-23T02:20:42.2534272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvc3f02gi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2534541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2534665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2535222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2535358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2535541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_e4v8xs
2022-11-23T02:20:42.2535735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_e4v8xs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2535898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2536060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2536221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2536379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2536435Z ok (2.577s)
2022-11-23T02:20:42.2536440Z 
2022-11-23T02:20:42.2536639Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2536713Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2536718Z 
2022-11-23T02:20:42.2536778Z OK
2022-11-23T02:20:42.2536783Z 
2022-11-23T02:20:42.2536918Z Generating XML reports...
2022-11-23T02:20:42.2537260Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020634.xml
2022-11-23T02:20:42.2537534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2537655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2537938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2538059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2538280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgyglayi
2022-11-23T02:20:42.2538472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgyglayi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2538735Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2538741Z 
2022-11-23T02:20:42.2538812Z Running tests...
2022-11-23T02:20:42.2539005Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2539238Z   test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27722
2022-11-23T02:20:42.2539395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27723
2022-11-23T02:20:42.2539537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27724
2022-11-23T02:20:42.2539694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27725
2022-11-23T02:20:42.2539967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2540088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2540369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2540503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2540686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb31v5wc
2022-11-23T02:20:42.2540878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb31v5wc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2541148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2541256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2541537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2541670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2541853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfutci9os
2022-11-23T02:20:42.2542047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfutci9os/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2542315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2542435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2542713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2542834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2543018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnebckcz
2022-11-23T02:20:42.2543210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnebckcz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2543478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2543631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2543908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2544039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2544219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50no37k3
2022-11-23T02:20:42.2544407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50no37k3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2544588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2544752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2544912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2545074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2545140Z ok (2.474s)
2022-11-23T02:20:42.2545146Z 
2022-11-23T02:20:42.2545340Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2545414Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2545419Z 
2022-11-23T02:20:42.2545478Z OK
2022-11-23T02:20:42.2545482Z 
2022-11-23T02:20:42.2545551Z Generating XML reports...
2022-11-23T02:20:42.2545895Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020638.xml
2022-11-23T02:20:42.2546167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2546290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2546569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2546704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2546885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5qhvbni
2022-11-23T02:20:42.2547078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5qhvbni/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2547329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2547334Z 
2022-11-23T02:20:42.2547393Z Running tests...
2022-11-23T02:20:42.2547588Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2547840Z   test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27945
2022-11-23T02:20:42.2547998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27946
2022-11-23T02:20:42.2548151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27947
2022-11-23T02:20:42.2548308Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27948
2022-11-23T02:20:42.2548582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2548702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2548984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2549105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2549291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppxkxghf
2022-11-23T02:20:42.2549490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppxkxghf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2549761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2549919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2550199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2550331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2550513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v7s6fot
2022-11-23T02:20:42.2550693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v7s6fot/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2550961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2551112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2551395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2551526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2551708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3cyvovw
2022-11-23T02:20:42.2551897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3cyvovw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2552164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2552284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2552549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2552682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2552863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3crfqddz
2022-11-23T02:20:42.2553058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3crfqddz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2553222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2553383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2553546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2553705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2553758Z ok (2.474s)
2022-11-23T02:20:42.2553775Z 
2022-11-23T02:20:42.2553959Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2554035Z Ran 1 test in 2.474s
2022-11-23T02:20:42.2554040Z 
2022-11-23T02:20:42.2554099Z OK
2022-11-23T02:20:42.2554104Z 
2022-11-23T02:20:42.2554187Z Generating XML reports...
2022-11-23T02:20:42.2554525Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020643.xml
2022-11-23T02:20:42.2554799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2554920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2555200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2555323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2555506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0zb3cgx
2022-11-23T02:20:42.2555699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0zb3cgx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2555954Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2555960Z 
2022-11-23T02:20:42.2556034Z Running tests...
2022-11-23T02:20:42.2556229Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2556486Z   test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28168
2022-11-23T02:20:42.2556641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28169
2022-11-23T02:20:42.2556781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28170
2022-11-23T02:20:42.2556933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28171
2022-11-23T02:20:42.2557206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2557357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2557638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2557773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2557958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7wgebpl
2022-11-23T02:20:42.2558153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7wgebpl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2558425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2558534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2558813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2558946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2559131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjm6eiefw
2022-11-23T02:20:42.2559397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2559516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2559797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2559929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2560099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco27btt9
2022-11-23T02:20:42.2560293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjm6eiefw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2560481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco27btt9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2560753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2560875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2561151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2561284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2561463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz9ypri4
2022-11-23T02:20:42.2561654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz9ypri4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2561803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2561964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2562127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2562288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2562353Z ok (5.584s)
2022-11-23T02:20:42.2562358Z 
2022-11-23T02:20:42.2562552Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2562656Z Ran 1 test in 5.584s
2022-11-23T02:20:42.2562662Z 
2022-11-23T02:20:42.2562722Z OK
2022-11-23T02:20:42.2562727Z 
2022-11-23T02:20:42.2562797Z Generating XML reports...
2022-11-23T02:20:42.2563132Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020647.xml
2022-11-23T02:20:42.2563405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2563529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2563811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2563997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2564180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps14f4cwr
2022-11-23T02:20:42.2564373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps14f4cwr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2564624Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2564630Z 
2022-11-23T02:20:42.2564690Z Running tests...
2022-11-23T02:20:42.2564884Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2565206Z   test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28391
2022-11-23T02:20:42.2565364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28392
2022-11-23T02:20:42.2565519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28393
2022-11-23T02:20:42.2565670Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28394
2022-11-23T02:20:42.2565948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2566072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2566354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2566476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2566659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hz5a2qz
2022-11-23T02:20:42.2566854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hz5a2qz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2567128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2567250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2567532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2567668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2567852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5_khuqw
2022-11-23T02:20:42.2568033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5_khuqw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2568307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2568427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2568708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2568845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2569027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v6g5j_o
2022-11-23T02:20:42.2569254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v6g5j_o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2569527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2569649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2569914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2570047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2570229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44boj3by
2022-11-23T02:20:42.2570453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44boj3by/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2570617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2570779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2570943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2571102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2571155Z ok (5.577s)
2022-11-23T02:20:42.2571172Z 
2022-11-23T02:20:42.2571356Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2571429Z Ran 1 test in 5.578s
2022-11-23T02:20:42.2571434Z 
2022-11-23T02:20:42.2571493Z OK
2022-11-23T02:20:42.2571497Z 
2022-11-23T02:20:42.2571579Z Generating XML reports...
2022-11-23T02:20:42.2571923Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020654.xml
2022-11-23T02:20:42.2572197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2572317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2572598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2572718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2572902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79wpwtkd
2022-11-23T02:20:42.2573098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79wpwtkd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2573351Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2573358Z 
2022-11-23T02:20:42.2573431Z Running tests...
2022-11-23T02:20:42.2573627Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2573860Z   test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28614
2022-11-23T02:20:42.2574017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28615
2022-11-23T02:20:42.2574168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28616
2022-11-23T02:20:42.2574308Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28617
2022-11-23T02:20:42.2574582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2574705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2575155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2575294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2575479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidb29oyy
2022-11-23T02:20:42.2575672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidb29oyy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2576002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2576111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2576391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2576524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2576709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqh1u9xkw
2022-11-23T02:20:42.2576903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqh1u9xkw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2577212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2577331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2577613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2577747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2577915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2trdf4en
2022-11-23T02:20:42.2578107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2trdf4en/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2578375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2578497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2578779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2578910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2579090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogi957gl
2022-11-23T02:20:42.2579282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogi957gl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2579432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2579595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2579755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2579916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2579985Z ok (2.477s)
2022-11-23T02:20:42.2579990Z 
2022-11-23T02:20:42.2580184Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2580258Z Ran 1 test in 2.477s
2022-11-23T02:20:42.2580263Z 
2022-11-23T02:20:42.2580321Z OK
2022-11-23T02:20:42.2580325Z 
2022-11-23T02:20:42.2580408Z Generating XML reports...
2022-11-23T02:20:42.2580736Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020702.xml
2022-11-23T02:20:42.2581007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2581131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2581411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2581543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2581726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w742_hv
2022-11-23T02:20:42.2581915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w742_hv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2582168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2582173Z 
2022-11-23T02:20:42.2582263Z Running tests...
2022-11-23T02:20:42.2582457Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2582707Z   test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28837
2022-11-23T02:20:42.2582865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28838
2022-11-23T02:20:42.2583020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28839
2022-11-23T02:20:42.2583171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28840
2022-11-23T02:20:42.2583479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2583601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2583885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2584009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2584193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5y034z4
2022-11-23T02:20:42.2584385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5y034z4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2584658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2584780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2585066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2585198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2585381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxw0436t
2022-11-23T02:20:42.2585563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxw0436t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2585831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2585952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2586228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2586362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2586548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsmjhkzg
2022-11-23T02:20:42.2586744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsmjhkzg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2587014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2587137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2587406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2587540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2587720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3b4qtc8
2022-11-23T02:20:42.2587913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3b4qtc8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2588076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2588240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2588400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2588559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2588647Z ok (2.576s)
2022-11-23T02:20:42.2588665Z 
2022-11-23T02:20:42.2588850Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2588923Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2588928Z 
2022-11-23T02:20:42.2588987Z OK
2022-11-23T02:20:42.2588992Z 
2022-11-23T02:20:42.2589074Z Generating XML reports...
2022-11-23T02:20:42.2589415Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020706.xml
2022-11-23T02:20:42.2589691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2589843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2590122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2590243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2590428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04_s4672
2022-11-23T02:20:42.2590619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04_s4672/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2590874Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2590879Z 
2022-11-23T02:20:42.2590949Z Running tests...
2022-11-23T02:20:42.2591144Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2591378Z   test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29060
2022-11-23T02:20:42.2591535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29061
2022-11-23T02:20:42.2591689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29062
2022-11-23T02:20:42.2591831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29063
2022-11-23T02:20:42.2592108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2592229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2592511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2592643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2592825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8wkvehw
2022-11-23T02:20:42.2593021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8wkvehw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2593293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2593403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2593684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2593818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2594000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52575az2
2022-11-23T02:20:42.2594190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52575az2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2594459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2594583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2594861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2594993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2595231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9aaxmaq
2022-11-23T02:20:42.2595427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9aaxmaq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2595696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2595816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2596096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2596258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2596441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzdi5fw8
2022-11-23T02:20:42.2596633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzdi5fw8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2596785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2596946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2597106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2597265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2597331Z ok (2.573s)
2022-11-23T02:20:42.2597336Z 
2022-11-23T02:20:42.2597534Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2597607Z Ran 1 test in 2.573s
2022-11-23T02:20:42.2597614Z 
2022-11-23T02:20:42.2597673Z OK
2022-11-23T02:20:42.2597677Z 
2022-11-23T02:20:42.2597759Z Generating XML reports...
2022-11-23T02:20:42.2598082Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020711.xml
2022-11-23T02:20:42.2598355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2598476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2598753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2598883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2599066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqeyrixq
2022-11-23T02:20:42.2599260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqeyrixq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2599514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2599519Z 
2022-11-23T02:20:42.2599590Z Running tests...
2022-11-23T02:20:42.2599773Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2600026Z   test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29283
2022-11-23T02:20:42.2600185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29284
2022-11-23T02:20:42.2600344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29285
2022-11-23T02:20:42.2600493Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29286
2022-11-23T02:20:42.2600767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2600891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2601172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2601293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2601505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve_xlqy_
2022-11-23T02:20:42.2601700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve_xlqy_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2601975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2602096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2602377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2602510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2602721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpvx0vmc
2022-11-23T02:20:42.2602917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpvx0vmc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2603177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2603301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2603579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2603714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2603898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxt1ouab
2022-11-23T02:20:42.2604090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxt1ouab/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2604364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2604484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2604750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2604885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2605160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76d46lou
2022-11-23T02:20:42.2605352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76d46lou/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2605518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2605678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2605839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2606000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2606065Z ok (2.574s)
2022-11-23T02:20:42.2606070Z 
2022-11-23T02:20:42.2606253Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2606326Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2606334Z 
2022-11-23T02:20:42.2606397Z OK
2022-11-23T02:20:42.2606402Z 
2022-11-23T02:20:42.2606485Z Generating XML reports...
2022-11-23T02:20:42.2606826Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020715.xml
2022-11-23T02:20:42.2607099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2607223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2607500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2607625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2607810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uhop8sh
2022-11-23T02:20:42.2608005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uhop8sh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2608298Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2608304Z 
2022-11-23T02:20:42.2608376Z Running tests...
2022-11-23T02:20:42.2608638Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2608867Z   test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29506
2022-11-23T02:20:42.2609022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29507
2022-11-23T02:20:42.2609215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29508
2022-11-23T02:20:42.2609356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29509
2022-11-23T02:20:42.2609629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2609755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2610035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2610168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2610350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5ugyece
2022-11-23T02:20:42.2610596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5ugyece/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2610913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2611024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2611301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2611436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2611623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvxymk52
2022-11-23T02:20:42.2611816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvxymk52/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2612085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2612204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2612481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2612615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2612784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslqw1chi
2022-11-23T02:20:42.2612977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslqw1chi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2613254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2613374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2613652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2613785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2613966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkg1eu3d8
2022-11-23T02:20:42.2614160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkg1eu3d8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2614310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2614471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2614664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2614825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2614997Z ok (2.480s)
2022-11-23T02:20:42.2615003Z 
2022-11-23T02:20:42.2615202Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2615274Z Ran 1 test in 2.480s
2022-11-23T02:20:42.2615280Z 
2022-11-23T02:20:42.2615339Z OK
2022-11-23T02:20:42.2615344Z 
2022-11-23T02:20:42.2615427Z Generating XML reports...
2022-11-23T02:20:42.2615755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020720.xml
2022-11-23T02:20:42.2616078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2616201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2616481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2616615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2616797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngdr3du3
2022-11-23T02:20:42.2616989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngdr3du3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2617244Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2617249Z 
2022-11-23T02:20:42.2617320Z Running tests...
2022-11-23T02:20:42.2617504Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2617758Z   test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29729
2022-11-23T02:20:42.2617915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29730
2022-11-23T02:20:42.2618071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29731
2022-11-23T02:20:42.2618222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29732
2022-11-23T02:20:42.2618496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2618617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2618901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2619025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2619209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsm_s4xk
2022-11-23T02:20:42.2619403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsm_s4xk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2619679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2619801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2620081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2620215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2620398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygv_2hz_
2022-11-23T02:20:42.2620589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygv_2hz_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2620850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2620970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2621279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2621415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2621597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mjl3msm
2022-11-23T02:20:42.2621788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mjl3msm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2622058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2622179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2622473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2622610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2622794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9nol1f_
2022-11-23T02:20:42.2622989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9nol1f_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2623153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2623315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2623476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2623634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2623700Z ok (2.475s)
2022-11-23T02:20:42.2623707Z 
2022-11-23T02:20:42.2623890Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2623965Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2623970Z 
2022-11-23T02:20:42.2624029Z OK
2022-11-23T02:20:42.2624034Z 
2022-11-23T02:20:42.2624117Z Generating XML reports...
2022-11-23T02:20:42.2624459Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020724.xml
2022-11-23T02:20:42.2624735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2624855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2625133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2625252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2625434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v4v6fbc
2022-11-23T02:20:42.2625628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v4v6fbc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2625881Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2625886Z 
2022-11-23T02:20:42.2625957Z Running tests...
2022-11-23T02:20:42.2626153Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2626383Z   test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29952
2022-11-23T02:20:42.2626537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29953
2022-11-23T02:20:42.2626689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29954
2022-11-23T02:20:42.2626828Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29955
2022-11-23T02:20:42.2627102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2627224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2627504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2627665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2627849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq_w0rkl
2022-11-23T02:20:42.2628042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq_w0rkl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2628312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2628422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2628700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2628860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2629044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj3wqp5t
2022-11-23T02:20:42.2629239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj3wqp5t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2629510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2629630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2629907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2630040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2630211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjya4ot4
2022-11-23T02:20:42.2630404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjya4ot4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2630672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2630796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2631076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2631207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2631391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmhsp63n
2022-11-23T02:20:42.2631583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmhsp63n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2631745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2631897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2632055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2632212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2632279Z ok (2.574s)
2022-11-23T02:20:42.2632284Z 
2022-11-23T02:20:42.2632483Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2632556Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2632561Z 
2022-11-23T02:20:42.2632619Z OK
2022-11-23T02:20:42.2632624Z 
2022-11-23T02:20:42.2632704Z Generating XML reports...
2022-11-23T02:20:42.2633031Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020728.xml
2022-11-23T02:20:42.2633301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2633424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2633701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2633837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2634054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhpii30v
2022-11-23T02:20:42.2634250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhpii30v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2634502Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2634508Z 
2022-11-23T02:20:42.2634579Z Running tests...
2022-11-23T02:20:42.2634762Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2635014Z   test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30175
2022-11-23T02:20:42.2635202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30176
2022-11-23T02:20:42.2635355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30177
2022-11-23T02:20:42.2635506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30178
2022-11-23T02:20:42.2635783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2635904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2636184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2636304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2636488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8v47n3z
2022-11-23T02:20:42.2636683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8v47n3z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2636958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2637079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2637360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2637492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2637673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jmz3yxv
2022-11-23T02:20:42.2637864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jmz3yxv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2638124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2638246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2638522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2638653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2638837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwjgvqke
2022-11-23T02:20:42.2639104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2639226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2639500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2639620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2639804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqq4rk3f
2022-11-23T02:20:42.2639998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwjgvqke/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2640191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqq4rk3f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2640354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2640544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2640705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2640865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2640932Z ok (2.577s)
2022-11-23T02:20:42.2640937Z 
2022-11-23T02:20:42.2641121Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2641195Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2641200Z 
2022-11-23T02:20:42.2641290Z OK
2022-11-23T02:20:42.2641295Z 
2022-11-23T02:20:42.2641380Z Generating XML reports...
2022-11-23T02:20:42.2641715Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020733.xml
2022-11-23T02:20:42.2641985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2642109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2642385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2642505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2642687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef9fjtn1
2022-11-23T02:20:42.2642881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef9fjtn1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2643139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2643144Z 
2022-11-23T02:20:42.2643214Z Running tests...
2022-11-23T02:20:42.2643406Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2643637Z   test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30398
2022-11-23T02:20:42.2643792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30399
2022-11-23T02:20:42.2643944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30400
2022-11-23T02:20:42.2644084Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30401
2022-11-23T02:20:42.2644359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2644481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2644763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2644960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2645148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjnqtvcs
2022-11-23T02:20:42.2645344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjnqtvcs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2645614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2645736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2645999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2646130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2646312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgt1ytcv
2022-11-23T02:20:42.2646504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgt1ytcv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2646775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2646931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2647215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2647347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2647516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4cd889m
2022-11-23T02:20:42.2647708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4cd889m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2647977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2648127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2648406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2648536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2648721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_bl3gd8
2022-11-23T02:20:42.2648911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_bl3gd8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2649073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2649222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2649381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2649542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2649612Z ok (4.478s)
2022-11-23T02:20:42.2649617Z 
2022-11-23T02:20:42.2649814Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2649888Z Ran 1 test in 4.478s
2022-11-23T02:20:42.2649894Z 
2022-11-23T02:20:42.2649952Z OK
2022-11-23T02:20:42.2649958Z 
2022-11-23T02:20:42.2650040Z Generating XML reports...
2022-11-23T02:20:42.2650364Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020737.xml
2022-11-23T02:20:42.2650636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2650757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2651032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2651169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2651353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb48yond0
2022-11-23T02:20:42.2651546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb48yond0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2651799Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2651804Z 
2022-11-23T02:20:42.2651875Z Running tests...
2022-11-23T02:20:42.2652056Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2652301Z   test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30621
2022-11-23T02:20:42.2652456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30622
2022-11-23T02:20:42.2652608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30623
2022-11-23T02:20:42.2652763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30624
2022-11-23T02:20:42.2653033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2653152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2653464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2653586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2653770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtvmchu7
2022-11-23T02:20:42.2653962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtvmchu7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2654233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2654408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2654685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2654818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2655113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntwwzzov
2022-11-23T02:20:42.2655309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntwwzzov/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2655570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2655692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2655966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2656097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2656275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbo7jwsh
2022-11-23T02:20:42.2656464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbo7jwsh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2656735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2656855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2657129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2657250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2657427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8owl5ixu
2022-11-23T02:20:42.2657619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8owl5ixu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2657782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2657943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2658104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2658264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2658328Z ok (4.577s)
2022-11-23T02:20:42.2658333Z 
2022-11-23T02:20:42.2658517Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2658588Z Ran 1 test in 4.577s
2022-11-23T02:20:42.2658593Z 
2022-11-23T02:20:42.2658650Z OK
2022-11-23T02:20:42.2658655Z 
2022-11-23T02:20:42.2658734Z Generating XML reports...
2022-11-23T02:20:42.2659070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020743.xml
2022-11-23T02:20:42.2659345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2659465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2659742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2659925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2660097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunn_etxz
2022-11-23T02:20:42.2660286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunn_etxz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2660539Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2660544Z 
2022-11-23T02:20:42.2660616Z Running tests...
2022-11-23T02:20:42.2660808Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2661060Z   test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30844
2022-11-23T02:20:42.2661214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30845
2022-11-23T02:20:42.2661401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30846
2022-11-23T02:20:42.2661593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30847
2022-11-23T02:20:42.2661870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2661989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2662269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2662401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2662588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw72uba41
2022-11-23T02:20:42.2662781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw72uba41/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2663050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2663173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2663439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2663601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2663808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15uwrrgv
2022-11-23T02:20:42.2664003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15uwrrgv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2664276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2664396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2664672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2664806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2664975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt41w27c3
2022-11-23T02:20:42.2665165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt41w27c3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2665434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2665555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2665830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2665964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2666146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd416s1r
2022-11-23T02:20:42.2666374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd416s1r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2666539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2666685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2666844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2667002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2667067Z ok (2.574s)
2022-11-23T02:20:42.2667072Z 
2022-11-23T02:20:42.2667268Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2667370Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2667375Z 
2022-11-23T02:20:42.2667433Z OK
2022-11-23T02:20:42.2667438Z 
2022-11-23T02:20:42.2667519Z Generating XML reports...
2022-11-23T02:20:42.2667847Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020750.xml
2022-11-23T02:20:42.2668125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2668248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2668529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2668666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2668849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2lo0v3x
2022-11-23T02:20:42.2669046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2lo0v3x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2669296Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2669301Z 
2022-11-23T02:20:42.2669372Z Running tests...
2022-11-23T02:20:42.2669556Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2669780Z   test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31067
2022-11-23T02:20:42.2669935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31068
2022-11-23T02:20:42.2670088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31069
2022-11-23T02:20:42.2670237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31070
2022-11-23T02:20:42.2670509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2670634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2670915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2671038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2671224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57ctijy7
2022-11-23T02:20:42.2671415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57ctijy7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2671687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2671808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2672152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2672349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2672567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_7nyi7a
2022-11-23T02:20:42.2672759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_7nyi7a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2673063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2673187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2673469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2673603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2673785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9khbfkn6
2022-11-23T02:20:42.2673979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9khbfkn6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2674281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2674401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2674671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2674806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2674985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyafmbrlc
2022-11-23T02:20:42.2675178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyafmbrlc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2675343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2675502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2675665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2675824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2675890Z ok (2.573s)
2022-11-23T02:20:42.2675895Z 
2022-11-23T02:20:42.2676082Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2676156Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2676161Z 
2022-11-23T02:20:42.2676221Z OK
2022-11-23T02:20:42.2676225Z 
2022-11-23T02:20:42.2676308Z Generating XML reports...
2022-11-23T02:20:42.2676646Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020754.xml
2022-11-23T02:20:42.2676920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2677042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2677321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2677444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2677629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpnddfbb
2022-11-23T02:20:42.2677827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpnddfbb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2678081Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2678086Z 
2022-11-23T02:20:42.2678158Z Running tests...
2022-11-23T02:20:42.2678354Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2678581Z   test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31290
2022-11-23T02:20:42.2678735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31291
2022-11-23T02:20:42.2678891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31292
2022-11-23T02:20:42.2679031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31293
2022-11-23T02:20:42.2679334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2679456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2679734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2679867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2680050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97xdpvy2
2022-11-23T02:20:42.2680244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97xdpvy2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2680541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2680662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2680931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2681065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2681250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0oj26n_
2022-11-23T02:20:42.2681443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0oj26n_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2681712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2681833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2682109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2682243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2682474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3op6jsi
2022-11-23T02:20:42.2682668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3op6jsi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2682938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2683060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2683339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2683471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2683652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecw4og3v
2022-11-23T02:20:42.2683844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecw4og3v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2684006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2684157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2684320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2684479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2684544Z ok (2.475s)
2022-11-23T02:20:42.2684550Z 
2022-11-23T02:20:42.2684744Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2684817Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2684822Z 
2022-11-23T02:20:42.2684881Z OK
2022-11-23T02:20:42.2684886Z 
2022-11-23T02:20:42.2685034Z Generating XML reports...
2022-11-23T02:20:42.2685382Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020759.xml
2022-11-23T02:20:42.2685654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2685775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2686084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2686218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2686401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8rhkb68
2022-11-23T02:20:42.2686594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8rhkb68/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2686844Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2686877Z 
2022-11-23T02:20:42.2692673Z Running tests...
2022-11-23T02:20:42.2692952Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2693295Z   test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31513
2022-11-23T02:20:42.2693595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31514
2022-11-23T02:20:42.2693894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31515
2022-11-23T02:20:42.2694156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31516
2022-11-23T02:20:42.2694511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2694686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2695178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2695356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2695653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jit5_5e
2022-11-23T02:20:42.2696026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jit5_5e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2696462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2696574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2696859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2696997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2697183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd1dn74b
2022-11-23T02:20:42.2697385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd1dn74b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2697658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2697782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2698066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2698202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2698373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrycsqu_
2022-11-23T02:20:42.2698569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrycsqu_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2698844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2698968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2699250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2699384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2699567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87fci451
2022-11-23T02:20:42.2699859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87fci451/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2700011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2700177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2700339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2700501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2700646Z ok (2.473s)
2022-11-23T02:20:42.2700652Z 
2022-11-23T02:20:42.2700857Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2700932Z Ran 1 test in 2.473s
2022-11-23T02:20:42.2700937Z 
2022-11-23T02:20:42.2700999Z OK
2022-11-23T02:20:42.2701003Z 
2022-11-23T02:20:42.2701086Z Generating XML reports...
2022-11-23T02:20:42.2701421Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020803.xml
2022-11-23T02:20:42.2701698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2701822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2702101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2702238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2702425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhddibrs
2022-11-23T02:20:42.2702620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhddibrs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2702876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2702881Z 
2022-11-23T02:20:42.2702954Z Running tests...
2022-11-23T02:20:42.2703136Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2703365Z   test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31736
2022-11-23T02:20:42.2703523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31737
2022-11-23T02:20:42.2703678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31738
2022-11-23T02:20:42.2703832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31739
2022-11-23T02:20:42.2704113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2704235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2704517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2704639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2704825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wfsaviy
2022-11-23T02:20:42.2705019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wfsaviy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2705294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2705416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2705702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2705836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2706020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyupae7g
2022-11-23T02:20:42.2706248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyupae7g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2706507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2706629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2706909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2707042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2707228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjk3uz6b
2022-11-23T02:20:42.2707467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjk3uz6b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2707738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2707860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2708127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2708258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2708441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofd2dbks
2022-11-23T02:20:42.2708635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofd2dbks/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2708801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2708966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2709127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2709288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2709344Z ok (2.475s)
2022-11-23T02:20:42.2709362Z 
2022-11-23T02:20:42.2709545Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2709620Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2709626Z 
2022-11-23T02:20:42.2709686Z OK
2022-11-23T02:20:42.2709691Z 
2022-11-23T02:20:42.2709774Z Generating XML reports...
2022-11-23T02:20:42.2710115Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020807.xml
2022-11-23T02:20:42.2710387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2710511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2710795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2710916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2711103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgn30ql9v
2022-11-23T02:20:42.2711298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgn30ql9v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2711558Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2711563Z 
2022-11-23T02:20:42.2711635Z Running tests...
2022-11-23T02:20:42.2711830Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2712055Z   test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31959
2022-11-23T02:20:42.2712217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31960
2022-11-23T02:20:42.2712370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31961
2022-11-23T02:20:42.2712510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31962
2022-11-23T02:20:42.2712813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2712937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2713221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2713356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2713541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpv48qxu
2022-11-23T02:20:42.2713762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpv48qxu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2714034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2714143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2714425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2714559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2714741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y2dyzl5
2022-11-23T02:20:42.2714936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y2dyzl5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2715205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2715330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2715610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2715743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2715914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0pljuwz
2022-11-23T02:20:42.2716110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0pljuwz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2716380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2716501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2716779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2716913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2717095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_61qfs37
2022-11-23T02:20:42.2717285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_61qfs37/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2717435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2717601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2717763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2717927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2718020Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2718210Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2718301Z Traceback (most recent call last):
2022-11-23T02:20:42.2718566Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2718688Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2718819Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2718824Z 
2022-11-23T02:20:42.2718914Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2719130Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2719222Z Traceback (most recent call last):
2022-11-23T02:20:42.2719481Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2719615Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2719745Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2719750Z 
2022-11-23T02:20:42.2719839Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2720041Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2720131Z Traceback (most recent call last):
2022-11-23T02:20:42.2720389Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2720518Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2720646Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2720650Z 
2022-11-23T02:20:42.2720740Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2720931Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2721021Z Traceback (most recent call last):
2022-11-23T02:20:42.2721263Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2721391Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2721515Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2721522Z 
2022-11-23T02:20:42.2721590Z ok (2.475s)
2022-11-23T02:20:42.2721595Z 
2022-11-23T02:20:42.2721795Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2721871Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2721876Z 
2022-11-23T02:20:42.2721935Z OK
2022-11-23T02:20:42.2721940Z 
2022-11-23T02:20:42.2722028Z Generating XML reports...
2022-11-23T02:20:42.2722358Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020812.xml
2022-11-23T02:20:42.2722633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2722757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2723041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2723178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2723366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ocuzp4m
2022-11-23T02:20:42.2723563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ocuzp4m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2723823Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2723828Z 
2022-11-23T02:20:42.2723902Z Running tests...
2022-11-23T02:20:42.2724085Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2724308Z   test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32182
2022-11-23T02:20:42.2724466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32183
2022-11-23T02:20:42.2724621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32184
2022-11-23T02:20:42.2724777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32185
2022-11-23T02:20:42.2725215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2725339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2725657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2725780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2725965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yt8mx85
2022-11-23T02:20:42.2726159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yt8mx85/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2726434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2726556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2726872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2727007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2727191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrtn52x1
2022-11-23T02:20:42.2727391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrtn52x1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2727649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2727771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2728051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2728184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2728371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr7aya1p
2022-11-23T02:20:42.2728564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr7aya1p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2728833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2728955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2729220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2729354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2729586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcnzqn5z
2022-11-23T02:20:42.2729858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcnzqn5z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2730135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2730444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2730741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2730917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2730987Z ok (2.475s)
2022-11-23T02:20:42.2730993Z 
2022-11-23T02:20:42.2731186Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2731260Z Ran 1 test in 2.476s
2022-11-23T02:20:42.2731266Z 
2022-11-23T02:20:42.2731328Z OK
2022-11-23T02:20:42.2731332Z 
2022-11-23T02:20:42.2731418Z Generating XML reports...
2022-11-23T02:20:42.2731756Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020816.xml
2022-11-23T02:20:42.2732029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2732153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2732433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2732600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2732788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqb52dmf
2022-11-23T02:20:42.2732984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqb52dmf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2733240Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2733246Z 
2022-11-23T02:20:42.2733318Z Running tests...
2022-11-23T02:20:42.2733513Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2733733Z   test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32405
2022-11-23T02:20:42.2733921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32406
2022-11-23T02:20:42.2734073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32407
2022-11-23T02:20:42.2734215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32408
2022-11-23T02:20:42.2734492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2734617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2734992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2735128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2735317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4_tiptk
2022-11-23T02:20:42.2735514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4_tiptk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2735787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2735911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2736181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2736314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2736498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lmigtx0
2022-11-23T02:20:42.2736690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lmigtx0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2736960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2737085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2737363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2737495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2737667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gk78z62
2022-11-23T02:20:42.2737859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gk78z62/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2738128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2738250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2738530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2738666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2738848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5omu7eh3
2022-11-23T02:20:42.2739038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5omu7eh3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2739255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2739407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2739568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2739729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2739796Z ok (2.476s)
2022-11-23T02:20:42.2739801Z 
2022-11-23T02:20:42.2740000Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2740111Z Ran 1 test in 2.477s
2022-11-23T02:20:42.2740116Z 
2022-11-23T02:20:42.2740175Z OK
2022-11-23T02:20:42.2740180Z 
2022-11-23T02:20:42.2740264Z Generating XML reports...
2022-11-23T02:20:42.2740591Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020820.xml
2022-11-23T02:20:42.2740867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2740990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2741267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2741399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2741583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn399y9v1
2022-11-23T02:20:42.2741774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn399y9v1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2742031Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2742036Z 
2022-11-23T02:20:42.2742107Z Running tests...
2022-11-23T02:20:42.2742292Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2742514Z   test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32628
2022-11-23T02:20:42.2742669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32629
2022-11-23T02:20:42.2742824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32630
2022-11-23T02:20:42.2742977Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32631
2022-11-23T02:20:42.2743249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2743373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2743653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2743773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2743960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0gqlms0
2022-11-23T02:20:42.2744154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0gqlms0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2744427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2744550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2744827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2744963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2745150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvwbg3ng
2022-11-23T02:20:42.2745344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvwbg3ng/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2745601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2745756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2746038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2746170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2746354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1aav4lvs
2022-11-23T02:20:42.2746542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1aav4lvs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2746843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2746964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2747229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2747364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2747547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_gl5t28
2022-11-23T02:20:42.2747737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_gl5t28/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2747901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2748065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2748226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2748388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2748453Z do nothing
2022-11-23T02:20:42.2748503Z do nothing
2022-11-23T02:20:42.2748565Z do nothing
2022-11-23T02:20:42.2748630Z do nothing
2022-11-23T02:20:42.2748692Z do nothing
2022-11-23T02:20:42.2748754Z do nothing
2022-11-23T02:20:42.2748819Z do nothing
2022-11-23T02:20:42.2748868Z do nothing
2022-11-23T02:20:42.2748933Z ok (2.573s)
2022-11-23T02:20:42.2748938Z 
2022-11-23T02:20:42.2749132Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2749206Z Ran 1 test in 2.573s
2022-11-23T02:20:42.2749211Z 
2022-11-23T02:20:42.2749270Z OK
2022-11-23T02:20:42.2749275Z 
2022-11-23T02:20:42.2749358Z Generating XML reports...
2022-11-23T02:20:42.2749695Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020825.xml
2022-11-23T02:20:42.2749969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2750078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2750357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2750493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2750678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe771jb_9
2022-11-23T02:20:42.2750873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe771jb_9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2751129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2751134Z 
2022-11-23T02:20:42.2751205Z Running tests...
2022-11-23T02:20:42.2751398Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2751621Z   test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 383
2022-11-23T02:20:42.2751762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 384
2022-11-23T02:20:42.2751912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 385
2022-11-23T02:20:42.2752092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 386
2022-11-23T02:20:42.2752367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2752490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2752772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2752904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2753133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt96mx0o
2022-11-23T02:20:42.2753313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt96mx0o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2753583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2753707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2753987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2754123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2754306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdmv_fy6
2022-11-23T02:20:42.2754498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdmv_fy6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2754768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2754890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2755154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2755288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2755471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz7zlw9n
2022-11-23T02:20:42.2755664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz7zlw9n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2755932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2756053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2756334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2756468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2756639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppaeiomuw
2022-11-23T02:20:42.2756830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppaeiomuw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2756995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2757158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2757319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2757481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2757657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2757834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2758008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2758168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2758494Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2758790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2759082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2759373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2759441Z ok (2.574s)
2022-11-23T02:20:42.2759474Z 
2022-11-23T02:20:42.2759670Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2759744Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2759750Z 
2022-11-23T02:20:42.2759811Z OK
2022-11-23T02:20:42.2759816Z 
2022-11-23T02:20:42.2759886Z Generating XML reports...
2022-11-23T02:20:42.2760230Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020829.xml
2022-11-23T02:20:42.2760503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2760624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2760903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2761037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2761219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05qzth24
2022-11-23T02:20:42.2761416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05qzth24/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2761668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2761673Z 
2022-11-23T02:20:42.2761732Z Running tests...
2022-11-23T02:20:42.2761931Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2762164Z   test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 618
2022-11-23T02:20:42.2762318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 619
2022-11-23T02:20:42.2762471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 620
2022-11-23T02:20:42.2762619Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 621
2022-11-23T02:20:42.2762898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2763021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2763290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2763427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2763612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo89bbq78
2022-11-23T02:20:42.2763809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo89bbq78/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2764086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2764259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2764600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2764735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2764991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpdxamer
2022-11-23T02:20:42.2765218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpdxamer/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2765495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2765619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2765899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2766035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2766219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3kvgrrg
2022-11-23T02:20:42.2766442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3kvgrrg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2766715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2766837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2767108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2767242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2767426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxud82em
2022-11-23T02:20:42.2767622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxud82em/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2767786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2767953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2768116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2768277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2768355Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2768571Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-11-23T02:20:42.2768664Z Traceback (most recent call last):
2022-11-23T02:20:42.2768927Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2769064Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2769362Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2022-11-23T02:20:42.2769462Z     raise ValueError(expected_err_escape)
2022-11-23T02:20:42.2769537Z ValueError: 
2022-11-23T02:20:42.2769599Z First line of error 
2022-11-23T02:20:42.2769673Z  next line of error 
2022-11-23T02:20:42.2769744Z  last line of error
2022-11-23T02:20:42.2769749Z 
2022-11-23T02:20:42.2769839Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2770045Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-11-23T02:20:42.2770138Z Traceback (most recent call last):
2022-11-23T02:20:42.2770399Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2770517Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2770810Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2022-11-23T02:20:42.2770910Z     raise ValueError(expected_err_escape)
2022-11-23T02:20:42.2770980Z ValueError: 
2022-11-23T02:20:42.2771057Z First line of error 
2022-11-23T02:20:42.2771129Z  next line of error 
2022-11-23T02:20:42.2771199Z  last line of error
2022-11-23T02:20:42.2771204Z 
2022-11-23T02:20:42.2771293Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2771486Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-11-23T02:20:42.2771574Z Traceback (most recent call last):
2022-11-23T02:20:42.2771873Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2772006Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2772295Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2022-11-23T02:20:42.2772394Z     raise ValueError(expected_err_escape)
2022-11-23T02:20:42.2772462Z ValueError: 
2022-11-23T02:20:42.2772523Z First line of error 
2022-11-23T02:20:42.2772595Z  next line of error 
2022-11-23T02:20:42.2772700Z  last line of error
2022-11-23T02:20:42.2772705Z 
2022-11-23T02:20:42.2772797Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2772998Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-11-23T02:20:42.2773089Z Traceback (most recent call last):
2022-11-23T02:20:42.2773345Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2773476Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2773751Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 417, in raise_func_escape
2022-11-23T02:20:42.2773850Z     raise ValueError(expected_err_escape)
2022-11-23T02:20:42.2773919Z ValueError: 
2022-11-23T02:20:42.2773993Z First line of error 
2022-11-23T02:20:42.2774068Z  next line of error 
2022-11-23T02:20:42.2774137Z  last line of error
2022-11-23T02:20:42.2774141Z 
2022-11-23T02:20:42.2774206Z ok (2.578s)
2022-11-23T02:20:42.2774213Z 
2022-11-23T02:20:42.2774399Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2774472Z Ran 1 test in 2.578s
2022-11-23T02:20:42.2774477Z 
2022-11-23T02:20:42.2774536Z OK
2022-11-23T02:20:42.2774541Z 
2022-11-23T02:20:42.2774625Z Generating XML reports...
2022-11-23T02:20:42.2775060Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020834.xml
2022-11-23T02:20:42.2775340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2775465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2775746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2775884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2776057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9dmw0ls
2022-11-23T02:20:42.2776259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9dmw0ls/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2776512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2776517Z 
2022-11-23T02:20:42.2776590Z Running tests...
2022-11-23T02:20:42.2776789Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2777009Z   test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 843
2022-11-23T02:20:42.2777163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 844
2022-11-23T02:20:42.2777312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 845
2022-11-23T02:20:42.2777449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 846
2022-11-23T02:20:42.2777726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2777852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2778133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2778314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2778502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7ggd868
2022-11-23T02:20:42.2778697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7ggd868/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2778971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2779095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2779363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2779534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2779719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulznliad
2022-11-23T02:20:42.2779918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulznliad/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2780192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2780314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2780594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2780728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2780894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ytb_04o
2022-11-23T02:20:42.2781089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ytb_04o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2781361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2781483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2781763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2781898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2782082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphez1g09o
2022-11-23T02:20:42.2782274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphez1g09o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2782438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2782588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2782753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2782915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2782982Z ok (2.574s)
2022-11-23T02:20:42.2782988Z 
2022-11-23T02:20:42.2783188Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2783263Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2783268Z 
2022-11-23T02:20:42.2783327Z OK
2022-11-23T02:20:42.2783331Z 
2022-11-23T02:20:42.2783415Z Generating XML reports...
2022-11-23T02:20:42.2783741Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020838.xml
2022-11-23T02:20:42.2784017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2784142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2784421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2784555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2784767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qjh1zau
2022-11-23T02:20:42.2784965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qjh1zau/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2785221Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2785226Z 
2022-11-23T02:20:42.2785298Z Running tests...
2022-11-23T02:20:42.2785479Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2785692Z   test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1066
2022-11-23T02:20:42.2785877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1067
2022-11-23T02:20:42.2786028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1068
2022-11-23T02:20:42.2786181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1069
2022-11-23T02:20:42.2786456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2786577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2786858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2786992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2787165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hw49jiy
2022-11-23T02:20:42.2787359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hw49jiy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2787630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2787756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2788037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2788169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2788351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04ruab3e
2022-11-23T02:20:42.2788542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04ruab3e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2788800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2788923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2789204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2789340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2789522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eb_xdd6
2022-11-23T02:20:42.2789714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eb_xdd6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2789983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2790104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2790379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2790499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2790680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u7cg3gj
2022-11-23T02:20:42.2790872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u7cg3gj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2791037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2791237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2791398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2791558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2791624Z ok (2.474s)
2022-11-23T02:20:42.2791629Z 
2022-11-23T02:20:42.2791812Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2791886Z Ran 1 test in 2.474s
2022-11-23T02:20:42.2791891Z 
2022-11-23T02:20:42.2791950Z OK
2022-11-23T02:20:42.2791954Z 
2022-11-23T02:20:42.2792068Z Generating XML reports...
2022-11-23T02:20:42.2792407Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020842.xml
2022-11-23T02:20:42.2792678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2792803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2793083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2793216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2793386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2c3e6vy
2022-11-23T02:20:42.2793579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2c3e6vy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2793832Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2793840Z 
2022-11-23T02:20:42.2793914Z Running tests...
2022-11-23T02:20:42.2794109Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2794335Z   test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1289
2022-11-23T02:20:42.2794491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1290
2022-11-23T02:20:42.2794644Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1291
2022-11-23T02:20:42.2794781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1292
2022-11-23T02:20:42.2795057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2795179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2795462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2795599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2795786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj35ovzyp
2022-11-23T02:20:42.2795984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj35ovzyp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2796256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2796378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2796646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2796783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2796968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bqq75h6
2022-11-23T02:20:42.2797165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bqq75h6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2797437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2797558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2797866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2798001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2798172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9_di5nn
2022-11-23T02:20:42.2798367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9_di5nn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2798636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2798799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2799079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2799213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2799399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk8odhh7
2022-11-23T02:20:42.2799593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk8odhh7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2799757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2799907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2800070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2800232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2800303Z ok (2.581s)
2022-11-23T02:20:42.2800309Z 
2022-11-23T02:20:42.2800507Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2800581Z Ran 1 test in 2.581s
2022-11-23T02:20:42.2800586Z 
2022-11-23T02:20:42.2800645Z OK
2022-11-23T02:20:42.2800650Z 
2022-11-23T02:20:42.2800733Z Generating XML reports...
2022-11-23T02:20:42.2801065Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020847.xml
2022-11-23T02:20:42.2801341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2801464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2801746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2801882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2802072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb51x1qsx
2022-11-23T02:20:42.2802267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb51x1qsx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2802523Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2802531Z 
2022-11-23T02:20:42.2802603Z Running tests...
2022-11-23T02:20:42.2802787Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2803003Z   test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1512
2022-11-23T02:20:42.2803158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1513
2022-11-23T02:20:42.2803312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1514
2022-11-23T02:20:42.2803463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1515
2022-11-23T02:20:42.2803740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2803861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2804222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2804359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2804532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b0fwjhs
2022-11-23T02:20:42.2804730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b0fwjhs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2805106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2805297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2805669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2805805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2805990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lqxuh_o
2022-11-23T02:20:42.2806191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lqxuh_o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2806450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2806573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2806854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2806989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2807176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3afdqko
2022-11-23T02:20:42.2807371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3afdqko/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2807644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2807768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2808046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2808167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2808350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf0lehxc
2022-11-23T02:20:42.2808546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf0lehxc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2808710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2808877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2809039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2809199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2809269Z ok (2.575s)
2022-11-23T02:20:42.2809274Z 
2022-11-23T02:20:42.2809456Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2809531Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2809536Z 
2022-11-23T02:20:42.2809596Z OK
2022-11-23T02:20:42.2809600Z 
2022-11-23T02:20:42.2809683Z Generating XML reports...
2022-11-23T02:20:42.2810025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020851.xml
2022-11-23T02:20:42.2810300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2810425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2810703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2810841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2811043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_96g_25x
2022-11-23T02:20:42.2811237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_96g_25x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2811492Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2811497Z 
2022-11-23T02:20:42.2811571Z Running tests...
2022-11-23T02:20:42.2811764Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2811990Z   test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1735
2022-11-23T02:20:42.2812176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1736
2022-11-23T02:20:42.2812328Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1737
2022-11-23T02:20:42.2812470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1738
2022-11-23T02:20:42.2812745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2812868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2813151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2813285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2813798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nb53h67
2022-11-23T02:20:42.2814000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nb53h67/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2814283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2814405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2814675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2814808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2815079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf1kgtgh
2022-11-23T02:20:42.2815276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf1kgtgh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2815549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2815675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2815957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2816094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2816266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o9nsigr
2022-11-23T02:20:42.2816460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o9nsigr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2816731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2816853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2817251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2817390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2817676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycanbud6
2022-11-23T02:20:42.2817872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycanbud6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2818101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2818254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2818420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2818584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2818653Z ok (2.475s)
2022-11-23T02:20:42.2818659Z 
2022-11-23T02:20:42.2818860Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2818935Z Ran 1 test in 2.475s
2022-11-23T02:20:42.2818975Z 
2022-11-23T02:20:42.2819036Z OK
2022-11-23T02:20:42.2819040Z 
2022-11-23T02:20:42.2819123Z Generating XML reports...
2022-11-23T02:20:42.2819451Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020856.xml
2022-11-23T02:20:42.2819728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2819851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2820133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2820269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2820452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nfrqd5i
2022-11-23T02:20:42.2820650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nfrqd5i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2820907Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2820913Z 
2022-11-23T02:20:42.2820986Z Running tests...
2022-11-23T02:20:42.2821169Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2821401Z   test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1958
2022-11-23T02:20:42.2821556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1959
2022-11-23T02:20:42.2821810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1960
2022-11-23T02:20:42.2821962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1961
2022-11-23T02:20:42.2822420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2822546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2822833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2822968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2823138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38ekmxit
2022-11-23T02:20:42.2823334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38ekmxit/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2823606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2823726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2824006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2824139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2824325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz2zwiwi
2022-11-23T02:20:42.2824518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz2zwiwi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2824775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2824928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2825210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2825342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2825525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsirlndzv
2022-11-23T02:20:42.2825719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsirlndzv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2826020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2826141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2826418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2826541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2826724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4ujzw3v
2022-11-23T02:20:42.2826916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4ujzw3v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2827079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2827243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2827405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2827568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2827636Z ok (2.677s)
2022-11-23T02:20:42.2827641Z 
2022-11-23T02:20:42.2827825Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2827900Z Ran 1 test in 2.677s
2022-11-23T02:20:42.2827905Z 
2022-11-23T02:20:42.2827965Z OK
2022-11-23T02:20:42.2827970Z 
2022-11-23T02:20:42.2828055Z Generating XML reports...
2022-11-23T02:20:42.2828399Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020900.xml
2022-11-23T02:20:42.2828674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2828796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2829173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2829311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2829482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebxj83bf
2022-11-23T02:20:42.2829677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebxj83bf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2829936Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2829941Z 
2022-11-23T02:20:42.2830014Z Running tests...
2022-11-23T02:20:42.2830209Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2830519Z   test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2181
2022-11-23T02:20:42.2830765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2182
2022-11-23T02:20:42.2830920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2183
2022-11-23T02:20:42.2831060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2184
2022-11-23T02:20:42.2831334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2831457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2831861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2831997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2832181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr1x8tm7
2022-11-23T02:20:42.2832375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr1x8tm7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2832649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2832801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2833068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2833203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2833388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvbvapcg
2022-11-23T02:20:42.2833701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvbvapcg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2833976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2834096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2834372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2834508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2834677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq4ha3t2
2022-11-23T02:20:42.2834868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq4ha3t2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2835140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2835262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2835543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2835676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2835857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_neiako9
2022-11-23T02:20:42.2836089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_neiako9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2836345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2836497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2836659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2836822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2837040Z ok (2.478s)
2022-11-23T02:20:42.2837045Z 
2022-11-23T02:20:42.2837243Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2837318Z Ran 1 test in 2.478s
2022-11-23T02:20:42.2837323Z 
2022-11-23T02:20:42.2837382Z OK
2022-11-23T02:20:42.2837387Z 
2022-11-23T02:20:42.2837470Z Generating XML reports...
2022-11-23T02:20:42.2837795Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020905.xml
2022-11-23T02:20:42.2838072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2838194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2838599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2838801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2839060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8p9b2wub
2022-11-23T02:20:42.2839256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8p9b2wub/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2839516Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2839522Z 
2022-11-23T02:20:42.2839594Z Running tests...
2022-11-23T02:20:42.2839778Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2840034Z   test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2404
2022-11-23T02:20:42.2840189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2405
2022-11-23T02:20:42.2840526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2406
2022-11-23T02:20:42.2840727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2407
2022-11-23T02:20:42.2841007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2841130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2841414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2841537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2841726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4o2uwzox
2022-11-23T02:20:42.2841922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4o2uwzox/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2842196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2842321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2842718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2842852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2843037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnnkrqi4
2022-11-23T02:20:42.2843233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnnkrqi4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2843497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2843620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2843898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2844035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2844220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30f9ryqe
2022-11-23T02:20:42.2844413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30f9ryqe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2844685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2844807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2845362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2845487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2845851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0xss4xh
2022-11-23T02:20:42.2846101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0xss4xh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2846265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2846429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2846591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2846753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2846821Z ok (2.576s)
2022-11-23T02:20:42.2846827Z 
2022-11-23T02:20:42.2847020Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2847137Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2847142Z 
2022-11-23T02:20:42.2847202Z OK
2022-11-23T02:20:42.2847207Z 
2022-11-23T02:20:42.2847291Z Generating XML reports...
2022-11-23T02:20:42.2847740Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020909.xml
2022-11-23T02:20:42.2848020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2848144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2848426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2848562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2848732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4_3n15k
2022-11-23T02:20:42.2848927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4_3n15k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2849183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2849188Z 
2022-11-23T02:20:42.2849260Z Running tests...
2022-11-23T02:20:42.2849460Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2849713Z   test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2627
2022-11-23T02:20:42.2849872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2628
2022-11-23T02:20:42.2850103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2629
2022-11-23T02:20:42.2850257Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2630
2022-11-23T02:20:42.2850535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2850663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2850943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2851081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2851266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdg1l25r
2022-11-23T02:20:42.2851466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdg1l25r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2851739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2851863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2852239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2852378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2852729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62a2fnwp
2022-11-23T02:20:42.2852927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62a2fnwp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2853311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2853439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2853722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2853856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2854027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1hq8g63
2022-11-23T02:20:42.2854399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1hq8g63/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2854675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2854997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2855285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2855427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2855611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz9izadw
2022-11-23T02:20:42.2855912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz9izadw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2856078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2856413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2856579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2856739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2856808Z ok (2.275s)
2022-11-23T02:20:42.2856814Z 
2022-11-23T02:20:42.2857016Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2857091Z Ran 1 test in 2.275s
2022-11-23T02:20:42.2857096Z 
2022-11-23T02:20:42.2857155Z OK
2022-11-23T02:20:42.2857160Z 
2022-11-23T02:20:42.2857248Z Generating XML reports...
2022-11-23T02:20:42.2857577Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020913.xml
2022-11-23T02:20:42.2857850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2857974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2858348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2858486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2858671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pf7ofee
2022-11-23T02:20:42.2858870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pf7ofee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2859322Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2859328Z 
2022-11-23T02:20:42.2859400Z Running tests...
2022-11-23T02:20:42.2859585Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2859794Z   test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2686
2022-11-23T02:20:42.2859949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2687
2022-11-23T02:20:42.2860101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2688
2022-11-23T02:20:42.2860253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2689
2022-11-23T02:20:42.2860585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2860712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2860991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2861113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2861300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bdfystp
2022-11-23T02:20:42.2861607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bdfystp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2862008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2862132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2862522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2862661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2862848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xp6juun
2022-11-23T02:20:42.2863041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xp6juun/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2863394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2863518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2863916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2864049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2864237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2h9oc5w
2022-11-23T02:20:42.2864431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2h9oc5w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2864703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2864825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2865101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2865222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2865408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7mh1b7k
2022-11-23T02:20:42.2865601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7mh1b7k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2865872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2866067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2866322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2866485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2866662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.2866825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.2866999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.2867308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2867482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.2867822Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2868120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2868621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.2868689Z ok (2.576s)
2022-11-23T02:20:42.2868695Z 
2022-11-23T02:20:42.2868929Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2869032Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2869058Z 
2022-11-23T02:20:42.2869183Z OK
2022-11-23T02:20:42.2869189Z 
2022-11-23T02:20:42.2869283Z Generating XML reports...
2022-11-23T02:20:42.2869631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020917.xml
2022-11-23T02:20:42.2869908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2870033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2870318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2870455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2870642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphpz9ob8g
2022-11-23T02:20:42.2870825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphpz9ob8g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2871083Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2871087Z 
2022-11-23T02:20:42.2871267Z Running tests...
2022-11-23T02:20:42.2871466Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2871996Z   test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2921
2022-11-23T02:20:42.2872270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2922
2022-11-23T02:20:42.2872541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2923
2022-11-23T02:20:42.2872804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2924
2022-11-23T02:20:42.2873296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2873501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2873786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2873920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2874106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd3i5w8o
2022-11-23T02:20:42.2874303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd3i5w8o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2874574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2874696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2874977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2875101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2875287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64e5yvce
2022-11-23T02:20:42.2875481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64e5yvce/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2875753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2876130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2876415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2876548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2876730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbq3z4loc
2022-11-23T02:20:42.2876924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbq3z4loc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2877181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2877332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2877612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2877746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2877931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jqn41kw
2022-11-23T02:20:42.2878121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jqn41kw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2878285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2878449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2878597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2878857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2878925Z ok (2.575s)
2022-11-23T02:20:42.2878931Z 
2022-11-23T02:20:42.2879131Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2879252Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2879257Z 
2022-11-23T02:20:42.2879317Z OK
2022-11-23T02:20:42.2879325Z 
2022-11-23T02:20:42.2879408Z Generating XML reports...
2022-11-23T02:20:42.2879763Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020922.xml
2022-11-23T02:20:42.2880407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2880519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2880897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2881037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2881221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi3kv5hc
2022-11-23T02:20:42.2881415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi3kv5hc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2881672Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2881678Z 
2022-11-23T02:20:42.2881751Z Running tests...
2022-11-23T02:20:42.2881947Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2882150Z   test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3144
2022-11-23T02:20:42.2882311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3145
2022-11-23T02:20:42.2882461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3146
2022-11-23T02:20:42.2882613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3147
2022-11-23T02:20:42.2882891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2883015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2883346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2883482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2883667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdih4ll9
2022-11-23T02:20:42.2883848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdih4ll9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2884121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2884275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2884559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2884694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2884882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasit0h0_
2022-11-23T02:20:42.2885256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasit0h0_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2885535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2885643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2886015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2886149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2886334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xv68vxb
2022-11-23T02:20:42.2886527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xv68vxb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2886799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2886927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2887204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2887338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2887508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1qpnk4i
2022-11-23T02:20:42.2887700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1qpnk4i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2887864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2888116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2888283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2888448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2888539Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2888662Z ValueError('Expected error')
2022-11-23T02:20:42.2888741Z Traceback (most recent call last):
2022-11-23T02:20:42.2889096Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2889232Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2889522Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2022-11-23T02:20:42.2889621Z     raise ValueError("Expected error")
2022-11-23T02:20:42.2889708Z ValueError: Expected error
2022-11-23T02:20:42.2889713Z 
2022-11-23T02:20:42.2889803Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2890050Z ValueError('Expected error')
2022-11-23T02:20:42.2890182Z Traceback (most recent call last):
2022-11-23T02:20:42.2890503Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2890640Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2890928Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2022-11-23T02:20:42.2891024Z     raise ValueError("Expected error")
2022-11-23T02:20:42.2891109Z ValueError: Expected error
2022-11-23T02:20:42.2891114Z 
2022-11-23T02:20:42.2891202Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2891354Z ValueError('Expected error')
2022-11-23T02:20:42.2891430Z Traceback (most recent call last):
2022-11-23T02:20:42.2891689Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2891822Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2892212Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2022-11-23T02:20:42.2892327Z     raise ValueError("Expected error")
2022-11-23T02:20:42.2892457Z ValueError: Expected error
2022-11-23T02:20:42.2892465Z 
2022-11-23T02:20:42.2892617Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2892757Z ValueError('Expected error')
2022-11-23T02:20:42.2892848Z Traceback (most recent call last):
2022-11-23T02:20:42.2893110Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2893242Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2893763Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 290, in raise_or_inc
2022-11-23T02:20:42.2893935Z     raise ValueError("Expected error")
2022-11-23T02:20:42.2894084Z ValueError: Expected error
2022-11-23T02:20:42.2894092Z 
2022-11-23T02:20:42.2894460Z ok (2.575s)
2022-11-23T02:20:42.2894469Z 
2022-11-23T02:20:42.2894750Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2894828Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2894833Z 
2022-11-23T02:20:42.2895031Z OK
2022-11-23T02:20:42.2895039Z 
2022-11-23T02:20:42.2895127Z Generating XML reports...
2022-11-23T02:20:42.2895479Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020926.xml
2022-11-23T02:20:42.2895757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2895885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2896166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2896301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2896478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a1s0w66
2022-11-23T02:20:42.2896672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a1s0w66/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2897037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2897043Z 
2022-11-23T02:20:42.2897115Z Running tests...
2022-11-23T02:20:42.2897315Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2897540Z   test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3367
2022-11-23T02:20:42.2897699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3368
2022-11-23T02:20:42.2897851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3369
2022-11-23T02:20:42.2898082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3370
2022-11-23T02:20:42.2898619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2898988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2899278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2899414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2899607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8stqlny
2022-11-23T02:20:42.2899809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8stqlny/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2900507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2900633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2900907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2901041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2901225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0m5z1lm9
2022-11-23T02:20:42.2901418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0m5z1lm9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2901692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2901816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2902097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2902232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2902417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvzuh82h
2022-11-23T02:20:42.2902757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvzuh82h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2903408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2903734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2904219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2904357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2904544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z2rmggi
2022-11-23T02:20:42.2904741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z2rmggi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2904955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2905258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2905423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2905586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2905678Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2905879Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2906074Z Traceback (most recent call last):
2022-11-23T02:20:42.2906342Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2906479Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2906594Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2906614Z 
2022-11-23T02:20:42.2906692Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.2906812Z ValueError('Expected error')
2022-11-23T02:20:42.2906957Z Traceback (most recent call last):
2022-11-23T02:20:42.2907215Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2907343Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2907626Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2907915Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2907992Z ValueError: Expected error
2022-11-23T02:20:42.2907998Z 
2022-11-23T02:20:42.2908128Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2908249Z ValueError('Expected error')
2022-11-23T02:20:42.2908339Z Traceback (most recent call last):
2022-11-23T02:20:42.2908596Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2908725Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2909005Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2909096Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2909168Z ValueError: Expected error
2022-11-23T02:20:42.2909173Z 
2022-11-23T02:20:42.2909263Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.2909452Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2909542Z Traceback (most recent call last):
2022-11-23T02:20:42.2909800Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2909932Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2910062Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2910068Z 
2022-11-23T02:20:42.2910156Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2910261Z ValueError('Expected error')
2022-11-23T02:20:42.2910354Z Traceback (most recent call last):
2022-11-23T02:20:42.2910608Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2910735Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2911171Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2911262Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2911347Z ValueError: Expected error
2022-11-23T02:20:42.2911353Z 
2022-11-23T02:20:42.2911443Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.2911619Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2911708Z Traceback (most recent call last):
2022-11-23T02:20:42.2912053Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2912185Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2912313Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2912318Z 
2022-11-23T02:20:42.2912405Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2912523Z ValueError('Expected error')
2022-11-23T02:20:42.2912600Z Traceback (most recent call last):
2022-11-23T02:20:42.2912852Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2912978Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2913263Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.2913353Z     raise ValueError(expected_err)
2022-11-23T02:20:42.2913438Z ValueError: Expected error
2022-11-23T02:20:42.2913443Z 
2022-11-23T02:20:42.2913532Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.2913758Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-11-23T02:20:42.2913836Z Traceback (most recent call last):
2022-11-23T02:20:42.2914096Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.2914225Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.2914353Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-11-23T02:20:42.2914358Z 
2022-11-23T02:20:42.2914426Z ok (2.674s)
2022-11-23T02:20:42.2914431Z 
2022-11-23T02:20:42.2914629Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2914762Z Ran 1 test in 2.675s
2022-11-23T02:20:42.2914766Z 
2022-11-23T02:20:42.2914827Z OK
2022-11-23T02:20:42.2914832Z 
2022-11-23T02:20:42.2915020Z Generating XML reports...
2022-11-23T02:20:42.2915583Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020931.xml
2022-11-23T02:20:42.2916177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2916304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2916586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2916721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2916906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiggy83zc
2022-11-23T02:20:42.2917101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiggy83zc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2917358Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2917363Z 
2022-11-23T02:20:42.2917434Z Running tests...
2022-11-23T02:20:42.2917619Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2917838Z   test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3590
2022-11-23T02:20:42.2918087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3591
2022-11-23T02:20:42.2918241Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3592
2022-11-23T02:20:42.2918392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3593
2022-11-23T02:20:42.2918667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2918793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2919073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2919195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2919381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzun6xzeh
2022-11-23T02:20:42.2919578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzun6xzeh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2919848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2920149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2920431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2920566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2920752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lqveizv
2022-11-23T02:20:42.2920945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lqveizv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2921249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2921374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2921651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2921787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2921975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9sys1cy
2022-11-23T02:20:42.2922167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9sys1cy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2922465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2922591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2922948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2923086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2923270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfoievgc
2022-11-23T02:20:42.2923464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfoievgc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2923628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2923793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2923958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2924121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2924189Z ok (2.480s)
2022-11-23T02:20:42.2924195Z 
2022-11-23T02:20:42.2924381Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2924457Z Ran 1 test in 2.480s
2022-11-23T02:20:42.2924463Z 
2022-11-23T02:20:42.2924523Z OK
2022-11-23T02:20:42.2924528Z 
2022-11-23T02:20:42.2924611Z Generating XML reports...
2022-11-23T02:20:42.2925130Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020935.xml
2022-11-23T02:20:42.2925412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2925534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2925817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2925941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2926126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jngf14n
2022-11-23T02:20:42.2926322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jngf14n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2926577Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2926582Z 
2022-11-23T02:20:42.2926653Z Running tests...
2022-11-23T02:20:42.2926847Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2927071Z   test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3813
2022-11-23T02:20:42.2927227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3814
2022-11-23T02:20:42.2927466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3815
2022-11-23T02:20:42.2927898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3816
2022-11-23T02:20:42.2928328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2928499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2928785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2928905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2929090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgtd5prl
2022-11-23T02:20:42.2929285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgtd5prl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2929557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2929714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2929994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2930129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2930311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_bblshv
2022-11-23T02:20:42.2930504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_bblshv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2931111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2931232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2931512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2931648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2931829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u5hgbrp
2022-11-23T02:20:42.2932020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u5hgbrp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2932383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2932505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2932876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2932999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2933181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd3espnz
2022-11-23T02:20:42.2933376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd3espnz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2933538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2933701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2933862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2934022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2934088Z ok (2.577s)
2022-11-23T02:20:42.2934094Z 
2022-11-23T02:20:42.2934282Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2934356Z Ran 1 test in 2.577s
2022-11-23T02:20:42.2934361Z 
2022-11-23T02:20:42.2934420Z OK
2022-11-23T02:20:42.2934425Z 
2022-11-23T02:20:42.2934507Z Generating XML reports...
2022-11-23T02:20:42.2934845Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020940.xml
2022-11-23T02:20:42.2935390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2935513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2936017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2936155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2936325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco5q0f1k
2022-11-23T02:20:42.2936520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco5q0f1k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2936776Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2936781Z 
2022-11-23T02:20:42.2936896Z Running tests...
2022-11-23T02:20:42.2937091Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2937314Z   test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4036
2022-11-23T02:20:42.2937466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4037
2022-11-23T02:20:42.2937619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4038
2022-11-23T02:20:42.2937756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4039
2022-11-23T02:20:42.2938029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2938151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2938559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2938716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2938899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7gl7eib
2022-11-23T02:20:42.2939090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7gl7eib/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2939364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2939486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2939751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2939885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2940067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lo8tibr
2022-11-23T02:20:42.2940367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lo8tibr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2940641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2940763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2941041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2941172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2941342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxz3nzsy
2022-11-23T02:20:42.2941534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxz3nzsy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2941801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2941925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2942204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2942374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2943122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxxi1bbl
2022-11-23T02:20:42.2943506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxxi1bbl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2943734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2943885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2944046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2944206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2944272Z ok (2.576s)
2022-11-23T02:20:42.2944313Z 
2022-11-23T02:20:42.2944521Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2944595Z Ran 1 test in 2.576s
2022-11-23T02:20:42.2944600Z 
2022-11-23T02:20:42.2944659Z OK
2022-11-23T02:20:42.2944665Z 
2022-11-23T02:20:42.2944747Z Generating XML reports...
2022-11-23T02:20:42.2945073Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020944.xml
2022-11-23T02:20:42.2945346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2945467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2945929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2946063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2946246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgkjz9dk
2022-11-23T02:20:42.2946444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgkjz9dk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2946893Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2946902Z 
2022-11-23T02:20:42.2947004Z Running tests...
2022-11-23T02:20:42.2947197Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2947548Z   test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4259
2022-11-23T02:20:42.2947706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4260
2022-11-23T02:20:42.2947856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4261
2022-11-23T02:20:42.2948010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4262
2022-11-23T02:20:42.2948290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2948410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2948778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2948903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2949089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10v0y_g5
2022-11-23T02:20:42.2949369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10v0y_g5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2949646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2949765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2950043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2950177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2950359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fzqg76u
2022-11-23T02:20:42.2950847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fzqg76u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2951324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2951450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2951732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2951866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2952050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7x2b6q0
2022-11-23T02:20:42.2952288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7x2b6q0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2952677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2952800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2953081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2953202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2953659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaovkzms0
2022-11-23T02:20:42.2954098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaovkzms0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2954267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2954434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2954594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2954753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2954820Z ok (2.573s)
2022-11-23T02:20:42.2954826Z 
2022-11-23T02:20:42.2955020Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2955096Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2955101Z 
2022-11-23T02:20:42.2955159Z OK
2022-11-23T02:20:42.2955164Z 
2022-11-23T02:20:42.2955247Z Generating XML reports...
2022-11-23T02:20:42.2955586Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020949.xml
2022-11-23T02:20:42.2955855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2955978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2956259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2956391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2956696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qtkx7ug
2022-11-23T02:20:42.2956972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qtkx7ug/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2957787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2957795Z 
2022-11-23T02:20:42.2957870Z Running tests...
2022-11-23T02:20:42.2958069Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2958294Z   test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4482
2022-11-23T02:20:42.2958452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4483
2022-11-23T02:20:42.2958604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4484
2022-11-23T02:20:42.2958741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4485
2022-11-23T02:20:42.2959257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2959600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2960104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2960391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2960986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15wyzeky
2022-11-23T02:20:42.2961334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15wyzeky/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2961873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2962092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2962579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2962807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2963009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqlw37j0
2022-11-23T02:20:42.2963308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqlw37j0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2963655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2963776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2964270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2964474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2964647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p49ckpr
2022-11-23T02:20:42.2964846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p49ckpr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2965191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2965314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2965593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2965731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2965917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp502vyvu8
2022-11-23T02:20:42.2966109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp502vyvu8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2966271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2966422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2966583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2966740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2966807Z ok (2.476s)
2022-11-23T02:20:42.2966814Z 
2022-11-23T02:20:42.2967011Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2967085Z Ran 1 test in 2.476s
2022-11-23T02:20:42.2967090Z 
2022-11-23T02:20:42.2967149Z OK
2022-11-23T02:20:42.2967154Z 
2022-11-23T02:20:42.2967238Z Generating XML reports...
2022-11-23T02:20:42.2967988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020953.xml
2022-11-23T02:20:42.2968476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2968867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2969233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2969368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2969552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3x75rj7
2022-11-23T02:20:42.2969747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3x75rj7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2970002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2970047Z 
2022-11-23T02:20:42.2970119Z Running tests...
2022-11-23T02:20:42.2970301Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2970530Z   test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4705
2022-11-23T02:20:42.2970686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4706
2022-11-23T02:20:42.2970838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4707
2022-11-23T02:20:42.2970987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4708
2022-11-23T02:20:42.2971602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2971791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2972190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2972315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2972610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_as61t2
2022-11-23T02:20:42.2972806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_as61t2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2973083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2973204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2973484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2973616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2973798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad0hz6bp
2022-11-23T02:20:42.2973994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad0hz6bp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2974253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2974374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2974654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2974787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2975110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7idbeqrp
2022-11-23T02:20:42.2975308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7idbeqrp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2975583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2975707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2975984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2976103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2976356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyhwiazo
2022-11-23T02:20:42.2976740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyhwiazo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2976905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2977067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2977229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2977390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2977500Z ok (2.575s)
2022-11-23T02:20:42.2977506Z 
2022-11-23T02:20:42.2977694Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2977767Z Ran 1 test in 2.575s
2022-11-23T02:20:42.2977772Z 
2022-11-23T02:20:42.2977831Z OK
2022-11-23T02:20:42.2977836Z 
2022-11-23T02:20:42.2977920Z Generating XML reports...
2022-11-23T02:20:42.2978259Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020957.xml
2022-11-23T02:20:42.2978678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2978801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2979176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2979311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2979487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplskaa3c_
2022-11-23T02:20:42.2979680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplskaa3c_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2979936Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2979941Z 
2022-11-23T02:20:42.2980013Z Running tests...
2022-11-23T02:20:42.2980207Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2980626Z   test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4940
2022-11-23T02:20:42.2980956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4941
2022-11-23T02:20:42.2981112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4942
2022-11-23T02:20:42.2981254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4943
2022-11-23T02:20:42.2981674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2981812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2982100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2982234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2982571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo5m7lb0
2022-11-23T02:20:42.2982888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo5m7lb0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2983788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2984111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2984392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2984786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2985112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjdbslkb
2022-11-23T02:20:42.2985519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjdbslkb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2986052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2986259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2986648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2986873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2987109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphafr55pg
2022-11-23T02:20:42.2987304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphafr55pg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2987581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2987705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2987986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2988120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2988307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryxlfkx8
2022-11-23T02:20:42.2988647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryxlfkx8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2988814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.2988964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.2989126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.2989288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.2989355Z ok (2.574s)
2022-11-23T02:20:42.2989362Z 
2022-11-23T02:20:42.2989564Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2989637Z Ran 1 test in 2.574s
2022-11-23T02:20:42.2989642Z 
2022-11-23T02:20:42.2989700Z OK
2022-11-23T02:20:42.2989705Z 
2022-11-23T02:20:42.2989788Z Generating XML reports...
2022-11-23T02:20:42.2990118Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021002.xml
2022-11-23T02:20:42.2990394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2990654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2991125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2991262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2991451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbnmx5g_
2022-11-23T02:20:42.2991659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbnmx5g_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2991915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.2991921Z 
2022-11-23T02:20:42.2991992Z Running tests...
2022-11-23T02:20:42.2992176Z ----------------------------------------------------------------------
2022-11-23T02:20:42.2992400Z   test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5163
2022-11-23T02:20:42.2992555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5164
2022-11-23T02:20:42.2992838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5165
2022-11-23T02:20:42.2993035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5166
2022-11-23T02:20:42.2993473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2993692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2993976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2994242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2994556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfbsdro7
2022-11-23T02:20:42.2995106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfbsdro7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2995711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2995918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2996403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2996623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2996937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bh709ue
2022-11-23T02:20:42.2997280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bh709ue/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2997763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2997893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2998177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2998309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2998495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn186u9m
2022-11-23T02:20:42.2998781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn186u9m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.2999109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.2999232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.2999513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.2999636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.2999820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphu0v_0yz
2022-11-23T02:20:42.3000010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphu0v_0yz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3000174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3000439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3000675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3000901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3000969Z ok (2.677s)
2022-11-23T02:20:42.3000974Z 
2022-11-23T02:20:42.3001164Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3001240Z Ran 1 test in 2.677s
2022-11-23T02:20:42.3001246Z 
2022-11-23T02:20:42.3001304Z OK
2022-11-23T02:20:42.3001309Z 
2022-11-23T02:20:42.3001392Z Generating XML reports...
2022-11-23T02:20:42.3001732Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021006.xml
2022-11-23T02:20:42.3002059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3002300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3002666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3002800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3002973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdxuh975
2022-11-23T02:20:42.3003167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdxuh975/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3003463Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3003468Z 
2022-11-23T02:20:42.3003538Z Running tests...
2022-11-23T02:20:42.3003732Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3004113Z   test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5386
2022-11-23T02:20:42.3004270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5387
2022-11-23T02:20:42.3004421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5388
2022-11-23T02:20:42.3004560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5389
2022-11-23T02:20:42.3004837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3005029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3005313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3005446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3005631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhct65ac
2022-11-23T02:20:42.3005824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhct65ac/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3006094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3006216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3006482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3006615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3006801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6xpoi8m
2022-11-23T02:20:42.3006995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6xpoi8m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3007265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3007387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3007921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3008055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3008227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2u38nbq
2022-11-23T02:20:42.3008418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2u38nbq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3008691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3008812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3009339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3009576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3009764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6_b2g8r
2022-11-23T02:20:42.3009958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6_b2g8r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3010120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3010269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3010429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3010641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3010816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3010995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3011300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3011472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3011769Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3012052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3012339Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3012629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3012695Z ok (3.575s)
2022-11-23T02:20:42.3012702Z 
2022-11-23T02:20:42.3012897Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3012971Z Ran 1 test in 3.575s
2022-11-23T02:20:42.3012977Z 
2022-11-23T02:20:42.3013035Z OK
2022-11-23T02:20:42.3013039Z 
2022-11-23T02:20:42.3013121Z Generating XML reports...
2022-11-23T02:20:42.3013462Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021011.xml
2022-11-23T02:20:42.3013820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3013932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3014216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3014351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3014535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61li8h3v
2022-11-23T02:20:42.3014730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61li8h3v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3015133Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3015140Z 
2022-11-23T02:20:42.3015213Z Running tests...
2022-11-23T02:20:42.3015409Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3015646Z   test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5621
2022-11-23T02:20:42.3015803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5622
2022-11-23T02:20:42.3015954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5623
2022-11-23T02:20:42.3016103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5624
2022-11-23T02:20:42.3016435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3016560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3016993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3017128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3017402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p0o3479
2022-11-23T02:20:42.3017665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3017975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3018257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3018389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3018573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cfpe7dg
2022-11-23T02:20:42.3018884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p0o3479/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3019206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cfpe7dg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3019490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3019600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3019882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3020016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3020198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeunbdxx
2022-11-23T02:20:42.3020394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeunbdxx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3020665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3020785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3021161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3021297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3021466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sjx1_u7
2022-11-23T02:20:42.3021657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sjx1_u7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3021819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3021983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3022143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3022301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3022478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3022655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3022815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3022988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3023291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3023628Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3023924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3024213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3024279Z ok (3.577s)
2022-11-23T02:20:42.3024285Z 
2022-11-23T02:20:42.3024482Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3024555Z Ran 1 test in 3.578s
2022-11-23T02:20:42.3024590Z 
2022-11-23T02:20:42.3024730Z OK
2022-11-23T02:20:42.3024748Z 
2022-11-23T02:20:42.3024858Z Generating XML reports...
2022-11-23T02:20:42.3025254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021016.xml
2022-11-23T02:20:42.3025535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3025658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3025940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3026075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3026262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoeos52gn
2022-11-23T02:20:42.3026458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoeos52gn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3026701Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3026718Z 
2022-11-23T02:20:42.3026778Z Running tests...
2022-11-23T02:20:42.3026971Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3027214Z   test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5857
2022-11-23T02:20:42.3027368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5858
2022-11-23T02:20:42.3027521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5859
2022-11-23T02:20:42.3027671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5860
2022-11-23T02:20:42.3027947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3028068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3028445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3028580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3028764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ni94oey
2022-11-23T02:20:42.3029099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ni94oey/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3029463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3029586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3030134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3030268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3030442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeky78b2d
2022-11-23T02:20:42.3030637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeky78b2d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3030915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3031084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3031364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3031497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3031678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7z9c_xrj
2022-11-23T02:20:42.3031868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7z9c_xrj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3032171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3032280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3032558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3032692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3033075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9zibcy1
2022-11-23T02:20:42.3033358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9zibcy1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3033523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3033684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3033846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3033995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3034061Z ok (2.574s)
2022-11-23T02:20:42.3034067Z 
2022-11-23T02:20:42.3034267Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3034339Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3034345Z 
2022-11-23T02:20:42.3034407Z OK
2022-11-23T02:20:42.3034412Z 
2022-11-23T02:20:42.3034494Z Generating XML reports...
2022-11-23T02:20:42.3034832Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021022.xml
2022-11-23T02:20:42.3035105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3035227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3035493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3035629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3035814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0swtta9
2022-11-23T02:20:42.3036008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0swtta9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3036261Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3036266Z 
2022-11-23T02:20:42.3036337Z Running tests...
2022-11-23T02:20:42.3036532Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3036748Z   test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6080
2022-11-23T02:20:42.3036903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6081
2022-11-23T02:20:42.3037041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6082
2022-11-23T02:20:42.3037194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6083
2022-11-23T02:20:42.3037466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3037587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3038103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3038240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3038423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qrxtb1y
2022-11-23T02:20:42.3038712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qrxtb1y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3039064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3039223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3039596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3039732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3039917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6tcekh2
2022-11-23T02:20:42.3040110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6tcekh2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3040382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3040503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3040782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3040905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3041086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl_jq49k
2022-11-23T02:20:42.3041588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl_jq49k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3041864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3041986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3042265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3042399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3042581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3_p632k
2022-11-23T02:20:42.3042760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3_p632k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3042923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3043086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3043246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3043498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3043564Z ok (2.477s)
2022-11-23T02:20:42.3043570Z 
2022-11-23T02:20:42.3043767Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3043936Z Ran 1 test in 2.478s
2022-11-23T02:20:42.3043944Z 
2022-11-23T02:20:42.3044021Z OK
2022-11-23T02:20:42.3044042Z 
2022-11-23T02:20:42.3044134Z Generating XML reports...
2022-11-23T02:20:42.3044479Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021026.xml
2022-11-23T02:20:42.3044754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3044874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3045260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3045395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3045578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrr1br88
2022-11-23T02:20:42.3045771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrr1br88/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3046011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3046028Z 
2022-11-23T02:20:42.3046087Z Running tests...
2022-11-23T02:20:42.3046279Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3046526Z   test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6303
2022-11-23T02:20:42.3046682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6304
2022-11-23T02:20:42.3046835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6305
2022-11-23T02:20:42.3046986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6306
2022-11-23T02:20:42.3047411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3047533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3047803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3047936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3048123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzxe13y0
2022-11-23T02:20:42.3048318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzxe13y0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3048589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3048712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3049083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3049275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3049477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwalyy89j
2022-11-23T02:20:42.3049857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwalyy89j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3050138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3050353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3050635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3050767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3051095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjimgwec2
2022-11-23T02:20:42.3051292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjimgwec2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3051565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3051674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3052045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3052397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3052583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogh4bf9l
2022-11-23T02:20:42.3052897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogh4bf9l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3053064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3053227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3053388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3053537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3053604Z ok (8.785s)
2022-11-23T02:20:42.3053609Z 
2022-11-23T02:20:42.3053808Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3053914Z Ran 1 test in 8.786s
2022-11-23T02:20:42.3053919Z 
2022-11-23T02:20:42.3053977Z OK
2022-11-23T02:20:42.3053982Z 
2022-11-23T02:20:42.3054067Z Generating XML reports...
2022-11-23T02:20:42.3054410Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021030.xml
2022-11-23T02:20:42.3054683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3054804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3055157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3055290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3055473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabhlzss9
2022-11-23T02:20:42.3055669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabhlzss9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3055921Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3055927Z 
2022-11-23T02:20:42.3055999Z Running tests...
2022-11-23T02:20:42.3056194Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3056656Z   test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6526
2022-11-23T02:20:42.3056800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6527
2022-11-23T02:20:42.3056951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6528
2022-11-23T02:20:42.3057101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6529
2022-11-23T02:20:42.3057376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3057501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3057785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3057923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3058110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn2mok2z
2022-11-23T02:20:42.3058382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3058556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3059038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3059260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3059574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm14bm0a5
2022-11-23T02:20:42.3059993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn2mok2z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3060197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm14bm0a5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3060549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3060673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3060952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3061073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3061256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfw9k2qp
2022-11-23T02:20:42.3061449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfw9k2qp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3062034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3062159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3062532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3062670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3062855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssxwop31
2022-11-23T02:20:42.3063133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssxwop31/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3063300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3063463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3063653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3063879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3064059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3064236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3064412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3064580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3064872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3065170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3065462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3065972Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3066038Z ok (2.775s)
2022-11-23T02:20:42.3066047Z 
2022-11-23T02:20:42.3066244Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3066318Z Ran 1 test in 2.776s
2022-11-23T02:20:42.3066323Z 
2022-11-23T02:20:42.3066383Z OK
2022-11-23T02:20:42.3066388Z 
2022-11-23T02:20:42.3066470Z Generating XML reports...
2022-11-23T02:20:42.3066798Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021041.xml
2022-11-23T02:20:42.3067070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3067195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3067475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3067773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3068263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jk_vw8n
2022-11-23T02:20:42.3068721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jk_vw8n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3069168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3069177Z 
2022-11-23T02:20:42.3069299Z Running tests...
2022-11-23T02:20:42.3069618Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3069996Z   test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6761
2022-11-23T02:20:42.3070309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6762
2022-11-23T02:20:42.3071096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6763
2022-11-23T02:20:42.3071359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6764
2022-11-23T02:20:42.3071796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3071921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3072203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3072327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3072515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyghlmpav
2022-11-23T02:20:42.3072713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyghlmpav/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3072984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3073104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3073381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3073677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3074186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk8r9d45
2022-11-23T02:20:42.3074503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk8r9d45/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3075219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3075551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3076046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3076280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3076583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf72ixr4m
2022-11-23T02:20:42.3076789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf72ixr4m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3077065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3077186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3077453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3077587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3077772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdkarube
2022-11-23T02:20:42.3077967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdkarube/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3078367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3078737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3078900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3079217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3079272Z ok (2.576s)
2022-11-23T02:20:42.3079291Z 
2022-11-23T02:20:42.3079481Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3079554Z Ran 1 test in 2.577s
2022-11-23T02:20:42.3079559Z 
2022-11-23T02:20:42.3079658Z OK
2022-11-23T02:20:42.3079663Z 
2022-11-23T02:20:42.3079746Z Generating XML reports...
2022-11-23T02:20:42.3080088Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021046.xml
2022-11-23T02:20:42.3080361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3080486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3080860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3081052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3081239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphma6s8ua
2022-11-23T02:20:42.3081569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphma6s8ua/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3081948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3081954Z 
2022-11-23T02:20:42.3082026Z Running tests...
2022-11-23T02:20:42.3082222Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3082442Z   test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6984
2022-11-23T02:20:42.3082600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6985
2022-11-23T02:20:42.3082751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6986
2022-11-23T02:20:42.3082992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6987
2022-11-23T02:20:42.3083273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3083396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3083679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3084073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3084392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1t0gmvq
2022-11-23T02:20:42.3084744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1t0gmvq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3085281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3085393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3085905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3086332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3086674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r4xvuhe
2022-11-23T02:20:42.3087204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r4xvuhe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3087697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3088292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3088803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3089032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3089341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpvzavdm
2022-11-23T02:20:42.3089774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpvzavdm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3090080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3090253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3090533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3090666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3090853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp562r9u2d
2022-11-23T02:20:42.3091047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp562r9u2d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3091197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3091361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3091624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3091900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3092009Z ok (2.577s)
2022-11-23T02:20:42.3092018Z 
2022-11-23T02:20:42.3092366Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3092489Z Ran 1 test in 2.577s
2022-11-23T02:20:42.3092497Z 
2022-11-23T02:20:42.3092592Z OK
2022-11-23T02:20:42.3092599Z 
2022-11-23T02:20:42.3092742Z Generating XML reports...
2022-11-23T02:20:42.3093311Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021050.xml
2022-11-23T02:20:42.3093597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3093720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3094004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3094139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3094324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsshhlbym
2022-11-23T02:20:42.3094520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsshhlbym/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3094775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3094780Z 
2022-11-23T02:20:42.3094840Z Running tests...
2022-11-23T02:20:42.3095349Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3095728Z   test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7207
2022-11-23T02:20:42.3096003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7208
2022-11-23T02:20:42.3096271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7209
2022-11-23T02:20:42.3096525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7210
2022-11-23T02:20:42.3096927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3097051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3097590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3097720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3097904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w_rdx49
2022-11-23T02:20:42.3098098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w_rdx49/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3098373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3098539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3098816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3098948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3099334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24xwtzvq
2022-11-23T02:20:42.3099517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24xwtzvq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3099790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3099911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3100189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3100526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3100715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5tujncv
2022-11-23T02:20:42.3100907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5tujncv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3101185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3101306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3101709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3101842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3102024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdptrngm
2022-11-23T02:20:42.3102218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdptrngm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3102383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3102547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3102706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3102865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3103029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3103202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3103374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3103545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3103851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3104149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3104442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3104772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3105489Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.3105712Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2022-11-23T02:20:42.3105719Z 
2022-11-23T02:20:42.3106259Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.3106443Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2022-11-23T02:20:42.3106448Z 
2022-11-23T02:20:42.3106978Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.3107335Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2022-11-23T02:20:42.3107345Z 
2022-11-23T02:20:42.3108179Z [W rref_context.cpp:164] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-11-23T02:20:42.3108364Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2022-11-23T02:20:42.3108369Z 
2022-11-23T02:20:42.3108435Z ok (2.576s)
2022-11-23T02:20:42.3108440Z 
2022-11-23T02:20:42.3108644Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3108709Z Ran 1 test in 2.576s
2022-11-23T02:20:42.3108714Z 
2022-11-23T02:20:42.3108774Z OK
2022-11-23T02:20:42.3108779Z 
2022-11-23T02:20:42.3108860Z Generating XML reports...
2022-11-23T02:20:42.3109205Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021054.xml
2022-11-23T02:20:42.3109482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3109605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3110100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3110370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3110549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmp63drw
2022-11-23T02:20:42.3110792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmp63drw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3111056Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3111061Z 
2022-11-23T02:20:42.3111134Z Running tests...
2022-11-23T02:20:42.3111333Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3111550Z   test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7442
2022-11-23T02:20:42.3111704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7443
2022-11-23T02:20:42.3111886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7444
2022-11-23T02:20:42.3112036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7445
2022-11-23T02:20:42.3112297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3112424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3112705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3112839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3113024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdj80hor
2022-11-23T02:20:42.3113218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdj80hor/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3113492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3113614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3113892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3114016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3114199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv8paj45
2022-11-23T02:20:42.3114391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv8paj45/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3114661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3114781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3115060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3115195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3115375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y_dajss
2022-11-23T02:20:42.3115554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y_dajss/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3115821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3115942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3116217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3116347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3116530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e3hnch0
2022-11-23T02:20:42.3116725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e3hnch0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3116887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3117049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3117226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3117389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3117455Z ok (2.674s)
2022-11-23T02:20:42.3117460Z 
2022-11-23T02:20:42.3117758Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3117832Z Ran 1 test in 2.674s
2022-11-23T02:20:42.3117838Z 
2022-11-23T02:20:42.3117897Z OK
2022-11-23T02:20:42.3117902Z 
2022-11-23T02:20:42.3118079Z Generating XML reports...
2022-11-23T02:20:42.3118473Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021059.xml
2022-11-23T02:20:42.3118825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3118950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3119235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3119370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3119553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisgjz8na
2022-11-23T02:20:42.3119748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisgjz8na/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3120005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3120012Z 
2022-11-23T02:20:42.3120084Z Running tests...
2022-11-23T02:20:42.3120279Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3120488Z   test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7665
2022-11-23T02:20:42.3120644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7666
2022-11-23T02:20:42.3120796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7667
2022-11-23T02:20:42.3120947Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7668
2022-11-23T02:20:42.3121220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3121345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3121797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3121934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3122105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jr30a27
2022-11-23T02:20:42.3122455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jr30a27/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3122938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3123153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3123478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3123612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3123797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_1uns87
2022-11-23T02:20:42.3123989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_1uns87/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3124262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3124372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3124690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3124825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3125076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkjl4zs8
2022-11-23T02:20:42.3125273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkjl4zs8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3125545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3125665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3125978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3126110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3126280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkic_oz6
2022-11-23T02:20:42.3126474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkic_oz6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3126637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3126798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3126958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3127118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3127187Z ok (2.675s)
2022-11-23T02:20:42.3127192Z 
2022-11-23T02:20:42.3127389Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3127451Z Ran 1 test in 2.676s
2022-11-23T02:20:42.3127457Z 
2022-11-23T02:20:42.3127517Z OK
2022-11-23T02:20:42.3127522Z 
2022-11-23T02:20:42.3127604Z Generating XML reports...
2022-11-23T02:20:42.3127944Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021103.xml
2022-11-23T02:20:42.3128215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3128336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3128615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3128749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3128923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1p26idj
2022-11-23T02:20:42.3129118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1p26idj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3129372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3129377Z 
2022-11-23T02:20:42.3129450Z Running tests...
2022-11-23T02:20:42.3129644Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3129865Z   test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7888
2022-11-23T02:20:42.3130018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7889
2022-11-23T02:20:42.3130369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7890
2022-11-23T02:20:42.3130542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7891
2022-11-23T02:20:42.3130812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3130935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3131212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3131384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3131569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q10eu7d
2022-11-23T02:20:42.3131761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q10eu7d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3132033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3132156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3132433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3132584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3132766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wd41y4e
2022-11-23T02:20:42.3132962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wd41y4e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3133234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3133355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3133630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3133764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3133948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpits89suh
2022-11-23T02:20:42.3134132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpits89suh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3134399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3134521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3134802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3135077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3135263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s9_drtn
2022-11-23T02:20:42.3135457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s9_drtn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3135621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3135786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3135937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3136097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3136162Z ok (2.574s)
2022-11-23T02:20:42.3136171Z 
2022-11-23T02:20:42.3136369Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3136442Z Ran 1 test in 2.574s
2022-11-23T02:20:42.3136448Z 
2022-11-23T02:20:42.3136507Z OK
2022-11-23T02:20:42.3136512Z 
2022-11-23T02:20:42.3136594Z Generating XML reports...
2022-11-23T02:20:42.3136935Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021108.xml
2022-11-23T02:20:42.3137195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3137319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3137596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3137729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3137968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwsgjvyo
2022-11-23T02:20:42.3138165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwsgjvyo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3138424Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3138429Z 
2022-11-23T02:20:42.3138500Z Running tests...
2022-11-23T02:20:42.3138694Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3138898Z   test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8111
2022-11-23T02:20:42.3139089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8112
2022-11-23T02:20:42.3139239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8113
2022-11-23T02:20:42.3139388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8114
2022-11-23T02:20:42.3139670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3139792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3140182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3140317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3140491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gu6ass3
2022-11-23T02:20:42.3140684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gu6ass3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3140960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3141083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3141368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3141501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3141682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7eoc662u
2022-11-23T02:20:42.3141873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7eoc662u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3142143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3142252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3142532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3142666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3142847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0nn9oo7
2022-11-23T02:20:42.3143042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0nn9oo7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3143312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3143434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3143711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3143831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3144015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplypn5va2
2022-11-23T02:20:42.3144207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplypn5va2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3144368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3144568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3144728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3144889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3144955Z ok (2.474s)
2022-11-23T02:20:42.3144960Z 
2022-11-23T02:20:42.3145157Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3145218Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3145223Z 
2022-11-23T02:20:42.3145280Z OK
2022-11-23T02:20:42.3145312Z 
2022-11-23T02:20:42.3145396Z Generating XML reports...
2022-11-23T02:20:42.3145733Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021112.xml
2022-11-23T02:20:42.3146004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3146127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3146404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3146537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3146707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq142t0wh
2022-11-23T02:20:42.3146899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq142t0wh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3147151Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3147158Z 
2022-11-23T02:20:42.3147231Z Running tests...
2022-11-23T02:20:42.3147425Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3147643Z   test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8334
2022-11-23T02:20:42.3147799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8335
2022-11-23T02:20:42.3147961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8336
2022-11-23T02:20:42.3148200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8337
2022-11-23T02:20:42.3148476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3148599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3148879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3149016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3149201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaepaxub6
2022-11-23T02:20:42.3149399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaepaxub6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3149674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3149793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3150074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3150198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3150381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4nuup6j
2022-11-23T02:20:42.3150579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4nuup6j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3150847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3150969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3151284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3151419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3151604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbvxn3em
2022-11-23T02:20:42.3151784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbvxn3em/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3152051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3152233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3152510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3152641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3152823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7dfa0oe
2022-11-23T02:20:42.3153017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7dfa0oe/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3153180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3153339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3153488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3153650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3153715Z ok (2.578s)
2022-11-23T02:20:42.3153721Z 
2022-11-23T02:20:42.3153916Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3153989Z Ran 1 test in 2.578s
2022-11-23T02:20:42.3153994Z 
2022-11-23T02:20:42.3154053Z OK
2022-11-23T02:20:42.3154058Z 
2022-11-23T02:20:42.3154142Z Generating XML reports...
2022-11-23T02:20:42.3154480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021117.xml
2022-11-23T02:20:42.3154740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3154861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3155141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3155279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3155460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o38dthg
2022-11-23T02:20:42.3155653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o38dthg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3155910Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3155915Z 
2022-11-23T02:20:42.3155984Z Running tests...
2022-11-23T02:20:42.3156178Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3156592Z   test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8557
2022-11-23T02:20:42.3156868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8558
2022-11-23T02:20:42.3157133Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8559
2022-11-23T02:20:42.3157380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8560
2022-11-23T02:20:42.3157667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3157786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3158116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3158250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3158423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvnh0lzd
2022-11-23T02:20:42.3158619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvnh0lzd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3158892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3159042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3159320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3159452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3159638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmylgnb0
2022-11-23T02:20:42.3159832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmylgnb0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3160101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3160208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3160486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3160617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3160803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdod5bny7
2022-11-23T02:20:42.3160997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdod5bny7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3161266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3161389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3161666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3161786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3161967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoel_wysp
2022-11-23T02:20:42.3162157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoel_wysp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3162322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3162483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3162646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3162807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3162873Z ok (2.573s)
2022-11-23T02:20:42.3162878Z 
2022-11-23T02:20:42.3163073Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3163135Z Ran 1 test in 2.574s
2022-11-23T02:20:42.3163140Z 
2022-11-23T02:20:42.3163199Z OK
2022-11-23T02:20:42.3163203Z 
2022-11-23T02:20:42.3163284Z Generating XML reports...
2022-11-23T02:20:42.3163622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021121.xml
2022-11-23T02:20:42.3163895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3164016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3164293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3164452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3164623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjxa3cyd
2022-11-23T02:20:42.3164816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjxa3cyd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3165142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3165148Z 
2022-11-23T02:20:42.3165218Z Running tests...
2022-11-23T02:20:42.3165413Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3165693Z   test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8780
2022-11-23T02:20:42.3165847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8781
2022-11-23T02:20:42.3165996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8782
2022-11-23T02:20:42.3166147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8783
2022-11-23T02:20:42.3166408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3166529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3166808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3166942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3167123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z9t6qgd
2022-11-23T02:20:42.3167318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z9t6qgd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3167587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3167910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3168282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3168404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3168586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7a4ktg9
2022-11-23T02:20:42.3168776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7a4ktg9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3169043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3169165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3169443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3169576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3169759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssmkddl9
2022-11-23T02:20:42.3169940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssmkddl9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3170206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3170325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3170600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3170735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3170914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf8ikouy
2022-11-23T02:20:42.3171105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf8ikouy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3171307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3171472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3171620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3171778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3171844Z ok (2.474s)
2022-11-23T02:20:42.3171849Z 
2022-11-23T02:20:42.3172043Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3172144Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3172149Z 
2022-11-23T02:20:42.3172207Z OK
2022-11-23T02:20:42.3172212Z 
2022-11-23T02:20:42.3172292Z Generating XML reports...
2022-11-23T02:20:42.3172629Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021126.xml
2022-11-23T02:20:42.3172892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3173012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3173293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3173427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3173610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu9_sdtv
2022-11-23T02:20:42.3173800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu9_sdtv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3174053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3174058Z 
2022-11-23T02:20:42.3174129Z Running tests...
2022-11-23T02:20:42.3174324Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3174522Z   test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9003
2022-11-23T02:20:42.3174675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9004
2022-11-23T02:20:42.3174826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9005
2022-11-23T02:20:42.3175100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9006
2022-11-23T02:20:42.3175382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3175507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3175790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3175922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3176100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv0r7xw5
2022-11-23T02:20:42.3176296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv0r7xw5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3176569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3176690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3176969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3177103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3177291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpnewejw
2022-11-23T02:20:42.3177485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpnewejw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3177813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3177925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3178203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3178337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3178518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6elozem
2022-11-23T02:20:42.3178711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6elozem/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3179020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3179143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3179421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3179741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3180009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3ulxo03
2022-11-23T02:20:42.3180205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3ulxo03/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3180367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3180529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3180688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3180849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3180915Z ok (2.574s)
2022-11-23T02:20:42.3180922Z 
2022-11-23T02:20:42.3181122Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3181185Z Ran 1 test in 2.574s
2022-11-23T02:20:42.3181191Z 
2022-11-23T02:20:42.3181249Z OK
2022-11-23T02:20:42.3181254Z 
2022-11-23T02:20:42.3181334Z Generating XML reports...
2022-11-23T02:20:42.3181672Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021130.xml
2022-11-23T02:20:42.3181946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3182069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3182347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3182482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3182654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu0yhtas
2022-11-23T02:20:42.3182850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu0yhtas/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3183105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3183111Z 
2022-11-23T02:20:42.3183182Z Running tests...
2022-11-23T02:20:42.3183377Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3183591Z   test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9226
2022-11-23T02:20:42.3183745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9227
2022-11-23T02:20:42.3183898Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9228
2022-11-23T02:20:42.3184045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9229
2022-11-23T02:20:42.3184306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3184468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3184750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3184884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3185066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dyl0kg6
2022-11-23T02:20:42.3185261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dyl0kg6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3185532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3185685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3185951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3186083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3186269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkfs03w_
2022-11-23T02:20:42.3186462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkfs03w_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3186728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3186849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3187125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3187260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3187443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2pwv_v5
2022-11-23T02:20:42.3187944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3188071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3188353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3188487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3188668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhxgzi59
2022-11-23T02:20:42.3188860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2pwv_v5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3189049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhxgzi59/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3189218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3189378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3189526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3189686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3189751Z ok (4.483s)
2022-11-23T02:20:42.3189756Z 
2022-11-23T02:20:42.3189951Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3190025Z Ran 1 test in 4.484s
2022-11-23T02:20:42.3190030Z 
2022-11-23T02:20:42.3190089Z OK
2022-11-23T02:20:42.3190093Z 
2022-11-23T02:20:42.3190175Z Generating XML reports...
2022-11-23T02:20:42.3190514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021134.xml
2022-11-23T02:20:42.3190775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3190898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3191332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3191556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3191845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9mg6fto
2022-11-23T02:20:42.3192042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9mg6fto/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3192301Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3192306Z 
2022-11-23T02:20:42.3192377Z Running tests...
2022-11-23T02:20:42.3192612Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3192822Z   test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9449
2022-11-23T02:20:42.3192974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9450
2022-11-23T02:20:42.3193128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9451
2022-11-23T02:20:42.3193276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9452
2022-11-23T02:20:42.3193547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3193665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3193946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3194079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3194256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop2q59gp
2022-11-23T02:20:42.3194450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop2q59gp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3194721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3194841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3195123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3195253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3195435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y194zby
2022-11-23T02:20:42.3195625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y194zby/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3195896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3196005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3196281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3196415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3196597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8hkp9ki
2022-11-23T02:20:42.3196789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8hkp9ki/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3197057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3197177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3197455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3197576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3197758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr2f11dc
2022-11-23T02:20:42.3197977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr2f11dc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3198141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3198303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3198461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3198617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3198683Z ok (2.573s)
2022-11-23T02:20:42.3198687Z 
2022-11-23T02:20:42.3198913Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3198974Z Ran 1 test in 2.573s
2022-11-23T02:20:42.3198979Z 
2022-11-23T02:20:42.3199040Z OK
2022-11-23T02:20:42.3199174Z 
2022-11-23T02:20:42.3199304Z Generating XML reports...
2022-11-23T02:20:42.3199736Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021141.xml
2022-11-23T02:20:42.3200010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3200132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3200412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3200546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3200721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8e5k1pv
2022-11-23T02:20:42.3200918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8e5k1pv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3201172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3201178Z 
2022-11-23T02:20:42.3201248Z Running tests...
2022-11-23T02:20:42.3201445Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3201667Z   test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9672
2022-11-23T02:20:42.3201822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9673
2022-11-23T02:20:42.3201971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9674
2022-11-23T02:20:42.3202123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9675
2022-11-23T02:20:42.3202383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3202505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3202786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3202925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3203108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8416ou28
2022-11-23T02:20:42.3203298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8416ou28/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3203571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3203691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3203959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3204094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3204276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqhii10o
2022-11-23T02:20:42.3204543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3204701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3205043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3205177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3205359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllw1c7ls
2022-11-23T02:20:42.3205552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqhii10o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3205768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllw1c7ls/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3206043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3206162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3206443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3206577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3206758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5cwv1cd
2022-11-23T02:20:42.3206953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5cwv1cd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3207116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3207278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3207428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3207589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3207654Z ok (2.575s)
2022-11-23T02:20:42.3207659Z 
2022-11-23T02:20:42.3207855Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3207927Z Ran 1 test in 2.576s
2022-11-23T02:20:42.3207933Z 
2022-11-23T02:20:42.3207992Z OK
2022-11-23T02:20:42.3207996Z 
2022-11-23T02:20:42.3208077Z Generating XML reports...
2022-11-23T02:20:42.3208415Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021145.xml
2022-11-23T02:20:42.3208676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3208796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3209077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3209213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3209397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9xvaihx
2022-11-23T02:20:42.3209590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9xvaihx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3209841Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3209846Z 
2022-11-23T02:20:42.3209917Z Running tests...
2022-11-23T02:20:42.3210112Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3210326Z   test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9895
2022-11-23T02:20:42.3210481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9896
2022-11-23T02:20:42.3210710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9897
2022-11-23T02:20:42.3210965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9898
2022-11-23T02:20:42.3211399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3211524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3211808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3211940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3212117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng1rshfj
2022-11-23T02:20:42.3212387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3212554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3212835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3212969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3213154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunebn6s9
2022-11-23T02:20:42.3213350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng1rshfj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3213540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunebn6s9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3213810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3213920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3214200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3214333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3214515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmwfmogs
2022-11-23T02:20:42.3214710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmwfmogs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3215147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3215273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3215551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3215672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3215855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgcpie5l
2022-11-23T02:20:42.3216050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgcpie5l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3216211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3216374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3216534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3216693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3216758Z ok (2.575s)
2022-11-23T02:20:42.3216764Z 
2022-11-23T02:20:42.3216959Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3217021Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3217026Z 
2022-11-23T02:20:42.3217083Z OK
2022-11-23T02:20:42.3217088Z 
2022-11-23T02:20:42.3217172Z Generating XML reports...
2022-11-23T02:20:42.3217514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021150.xml
2022-11-23T02:20:42.3217786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3217963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3218247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3218382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3218552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqomns6mx
2022-11-23T02:20:42.3218747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqomns6mx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3219000Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3219040Z 
2022-11-23T02:20:42.3219113Z Running tests...
2022-11-23T02:20:42.3219308Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3219536Z   test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10118
2022-11-23T02:20:42.3219693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10119
2022-11-23T02:20:42.3220094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10120
2022-11-23T02:20:42.3220340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10121
2022-11-23T02:20:42.3220611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3220732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3221014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3221149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3221329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoffa_ez0
2022-11-23T02:20:42.3221525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoffa_ez0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3221800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3221921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3222186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3222319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3222501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70ednkgg
2022-11-23T02:20:42.3222696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70ednkgg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3222962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3228635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3228984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3229125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3229315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs7c8tk_
2022-11-23T02:20:42.3229513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs7c8tk_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3229791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3229915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3230196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3230318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3230577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_18czkf0
2022-11-23T02:20:42.3230774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_18czkf0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3230937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3231099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3231260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3231421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3231526Z ok (2.475s)
2022-11-23T02:20:42.3231532Z 
2022-11-23T02:20:42.3231721Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3231798Z Ran 1 test in 2.476s
2022-11-23T02:20:42.3231803Z 
2022-11-23T02:20:42.3231862Z OK
2022-11-23T02:20:42.3231867Z 
2022-11-23T02:20:42.3231949Z Generating XML reports...
2022-11-23T02:20:42.3232292Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021154.xml
2022-11-23T02:20:42.3232565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3232688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3232971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3233104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3233279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsd0yzqqg
2022-11-23T02:20:42.3233475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsd0yzqqg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3233733Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3233741Z 
2022-11-23T02:20:42.3233811Z Running tests...
2022-11-23T02:20:42.3234009Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3234229Z   test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10341
2022-11-23T02:20:42.3234384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10342
2022-11-23T02:20:42.3234538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10343
2022-11-23T02:20:42.3234677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10344
2022-11-23T02:20:42.3234950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3235069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3235554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3235774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3236090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfey6rmvr
2022-11-23T02:20:42.3236355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfey6rmvr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3236635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3236756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3237025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3237161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3237343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuit6vxb
2022-11-23T02:20:42.3237577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuit6vxb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3237848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3237969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3238245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3238378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3238579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ilxhn8j
2022-11-23T02:20:42.3238773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ilxhn8j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3239045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3239168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3239445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3239580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3239764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2gbrue6
2022-11-23T02:20:42.3239956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2gbrue6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3240118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3240271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3240436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3240599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3240666Z ok (3.476s)
2022-11-23T02:20:42.3240671Z 
2022-11-23T02:20:42.3240868Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3240980Z Ran 1 test in 3.477s
2022-11-23T02:20:42.3240985Z 
2022-11-23T02:20:42.3241041Z OK
2022-11-23T02:20:42.3241047Z 
2022-11-23T02:20:42.3241128Z Generating XML reports...
2022-11-23T02:20:42.3241456Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021158.xml
2022-11-23T02:20:42.3241727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3241852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3242130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3242264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3242448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpqozilj
2022-11-23T02:20:42.3242641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpqozilj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3242894Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3242899Z 
2022-11-23T02:20:42.3242969Z Running tests...
2022-11-23T02:20:42.3243151Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3243383Z   test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10564
2022-11-23T02:20:42.3243540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10565
2022-11-23T02:20:42.3243693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10566
2022-11-23T02:20:42.3243901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10567
2022-11-23T02:20:42.3244175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3244296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3244580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3244701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3245085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9d9psu5
2022-11-23T02:20:42.3245477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9d9psu5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3245976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3246189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3246593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3246728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3246910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbow7lrd_
2022-11-23T02:20:42.3247104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbow7lrd_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3247365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3247490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3247770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3247904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3248089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal4ysl2h
2022-11-23T02:20:42.3248282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal4ysl2h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3248552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3248675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3248954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3249076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3249258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdedbjcr
2022-11-23T02:20:42.3249453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdedbjcr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3249617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3249779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3249937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3250097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3250187Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3250295Z ValueError('Expected error')
2022-11-23T02:20:42.3250385Z Traceback (most recent call last):
2022-11-23T02:20:42.3250651Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3250786Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3251068Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3251201Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3251288Z ValueError: Expected error
2022-11-23T02:20:42.3251294Z 
2022-11-23T02:20:42.3251383Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3252237Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3252369Z Traceback (most recent call last):
2022-11-23T02:20:42.3252631Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3252765Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3253030Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3253129Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3253393Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3253461Z     raise exc
2022-11-23T02:20:42.3253572Z ValueError: On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3253680Z ValueError('Expected error')
2022-11-23T02:20:42.3253772Z Traceback (most recent call last):
2022-11-23T02:20:42.3254029Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3254162Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3254442Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3254532Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3254617Z ValueError: Expected error
2022-11-23T02:20:42.3254623Z 
2022-11-23T02:20:42.3254628Z 
2022-11-23T02:20:42.3254716Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3254822Z ValueError('Expected error')
2022-11-23T02:20:42.3255046Z Traceback (most recent call last):
2022-11-23T02:20:42.3255309Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3255439Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3255719Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3255812Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3255897Z ValueError: Expected error
2022-11-23T02:20:42.3255902Z 
2022-11-23T02:20:42.3255989Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3256851Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3256932Z Traceback (most recent call last):
2022-11-23T02:20:42.3257192Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3257326Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3257590Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3257688Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3258180Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3258302Z     raise exc
2022-11-23T02:20:42.3258484Z ValueError: On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3258680Z ValueError('Expected error')
2022-11-23T02:20:42.3258838Z Traceback (most recent call last):
2022-11-23T02:20:42.3259294Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3259523Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3260035Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3260245Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3260387Z ValueError: Expected error
2022-11-23T02:20:42.3260397Z 
2022-11-23T02:20:42.3260404Z 
2022-11-23T02:20:42.3260543Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3260663Z ValueError('Expected error')
2022-11-23T02:20:42.3260754Z Traceback (most recent call last):
2022-11-23T02:20:42.3261012Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3261143Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3261419Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3261511Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3261591Z ValueError: Expected error
2022-11-23T02:20:42.3261596Z 
2022-11-23T02:20:42.3261684Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3262550Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3262644Z Traceback (most recent call last):
2022-11-23T02:20:42.3262889Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3263019Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3263280Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3263374Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3263636Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3263701Z     raise exc
2022-11-23T02:20:42.3263808Z ValueError: On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3263914Z ValueError('Expected error')
2022-11-23T02:20:42.3264005Z Traceback (most recent call last):
2022-11-23T02:20:42.3264261Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3264392Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3264673Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3264760Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3264840Z ValueError: Expected error
2022-11-23T02:20:42.3264845Z 
2022-11-23T02:20:42.3264850Z 
2022-11-23T02:20:42.3264937Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3265044Z ValueError('Expected error')
2022-11-23T02:20:42.3265133Z Traceback (most recent call last):
2022-11-23T02:20:42.3265387Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3265518Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3265861Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3265953Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3266035Z ValueError: Expected error
2022-11-23T02:20:42.3266039Z 
2022-11-23T02:20:42.3266127Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3266980Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3267114Z Traceback (most recent call last):
2022-11-23T02:20:42.3267360Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3267491Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3267757Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3267853Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3268119Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3268186Z     raise exc
2022-11-23T02:20:42.3268294Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3268413Z ValueError('Expected error')
2022-11-23T02:20:42.3268491Z Traceback (most recent call last):
2022-11-23T02:20:42.3268747Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3268877Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3269155Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3269245Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3269330Z ValueError: Expected error
2022-11-23T02:20:42.3269335Z 
2022-11-23T02:20:42.3269340Z 
2022-11-23T02:20:42.3269404Z ok (2.474s)
2022-11-23T02:20:42.3269410Z 
2022-11-23T02:20:42.3269610Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3269672Z Ran 1 test in 2.475s
2022-11-23T02:20:42.3269676Z 
2022-11-23T02:20:42.3269735Z OK
2022-11-23T02:20:42.3269739Z 
2022-11-23T02:20:42.3269821Z Generating XML reports...
2022-11-23T02:20:42.3270162Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021204.xml
2022-11-23T02:20:42.3270435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3270559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3270841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3270974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3271147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuu_j7c5
2022-11-23T02:20:42.3271341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuu_j7c5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3271594Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3271601Z 
2022-11-23T02:20:42.3271671Z Running tests...
2022-11-23T02:20:42.3271863Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3272275Z   test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10787
2022-11-23T02:20:42.3272593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10788
2022-11-23T02:20:42.3272873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10789
2022-11-23T02:20:42.3273124Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10790
2022-11-23T02:20:42.3273399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3273522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3273805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3273977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3274161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlzn1p68
2022-11-23T02:20:42.3274356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlzn1p68/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3274629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3274751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3275025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3275145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3275328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp389c3kj0
2022-11-23T02:20:42.3275518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp389c3kj0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3275788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3275908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3276388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3276621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3276947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6h6o9i8i
2022-11-23T02:20:42.3277270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6h6o9i8i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3277765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3277934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3278217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3278350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3278535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82ymw79s
2022-11-23T02:20:42.3278727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82ymw79s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3278889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3279050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3279200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3279359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3279451Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3279571Z ValueError('Expected error')
2022-11-23T02:20:42.3279662Z Traceback (most recent call last):
2022-11-23T02:20:42.3279923Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3280096Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3280381Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3280458Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3280540Z ValueError: Expected error
2022-11-23T02:20:42.3280547Z 
2022-11-23T02:20:42.3280633Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3281495Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3281621Z Traceback (most recent call last):
2022-11-23T02:20:42.3281886Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3282017Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3282282Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3282378Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3282629Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3282693Z     raise exc
2022-11-23T02:20:42.3282800Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3282920Z ValueError('Expected error')
2022-11-23T02:20:42.3283009Z Traceback (most recent call last):
2022-11-23T02:20:42.3283266Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3283397Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3283681Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3283758Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3283842Z ValueError: Expected error
2022-11-23T02:20:42.3283847Z 
2022-11-23T02:20:42.3283851Z 
2022-11-23T02:20:42.3283939Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3284055Z ValueError('Expected error')
2022-11-23T02:20:42.3284146Z Traceback (most recent call last):
2022-11-23T02:20:42.3284397Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3284525Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3284803Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3284880Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3285050Z ValueError: Expected error
2022-11-23T02:20:42.3285056Z 
2022-11-23T02:20:42.3285144Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3286003Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3286097Z Traceback (most recent call last):
2022-11-23T02:20:42.3286359Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3286490Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3286784Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3286883Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3287134Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3287200Z     raise exc
2022-11-23T02:20:42.3287307Z ValueError: On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3287427Z ValueError('Expected error')
2022-11-23T02:20:42.3287517Z Traceback (most recent call last):
2022-11-23T02:20:42.3287773Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3287933Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3288211Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3288289Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3288371Z ValueError: Expected error
2022-11-23T02:20:42.3288378Z 
2022-11-23T02:20:42.3288382Z 
2022-11-23T02:20:42.3288470Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3288588Z ValueError('Expected error')
2022-11-23T02:20:42.3288677Z Traceback (most recent call last):
2022-11-23T02:20:42.3288929Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3289056Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3289329Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3289408Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3289490Z ValueError: Expected error
2022-11-23T02:20:42.3289495Z 
2022-11-23T02:20:42.3289580Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3290896Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3290999Z Traceback (most recent call last):
2022-11-23T02:20:42.3291266Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3291396Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3291664Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3291763Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3292027Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3292080Z     raise exc
2022-11-23T02:20:42.3292190Z ValueError: On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3292309Z ValueError('Expected error')
2022-11-23T02:20:42.3292398Z Traceback (most recent call last):
2022-11-23T02:20:42.3292650Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3292778Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3293058Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3293139Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3293224Z ValueError: Expected error
2022-11-23T02:20:42.3293229Z 
2022-11-23T02:20:42.3293233Z 
2022-11-23T02:20:42.3293321Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3293439Z ValueError('Expected error')
2022-11-23T02:20:42.3293528Z Traceback (most recent call last):
2022-11-23T02:20:42.3293830Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3294147Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3294429Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3294506Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3294591Z ValueError: Expected error
2022-11-23T02:20:42.3294596Z 
2022-11-23T02:20:42.3294685Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3295687Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-11-23T02:20:42.3295842Z Traceback (most recent call last):
2022-11-23T02:20:42.3296104Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3296236Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3296499Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 435, in _rref_typeof_on_owner
2022-11-23T02:20:42.3296595Z     rref_type = type(rref.local_value())
2022-11-23T02:20:42.3296862Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:20:42.3296918Z     raise exc
2022-11-23T02:20:42.3297026Z ValueError: On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3297145Z ValueError('Expected error')
2022-11-23T02:20:42.3297236Z Traceback (most recent call last):
2022-11-23T02:20:42.3297498Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3297633Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3297913Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3297990Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3298075Z ValueError: Expected error
2022-11-23T02:20:42.3298081Z 
2022-11-23T02:20:42.3298085Z 
2022-11-23T02:20:42.3298240Z ok (2.576s)
2022-11-23T02:20:42.3298245Z 
2022-11-23T02:20:42.3298446Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3298523Z Ran 1 test in 2.577s
2022-11-23T02:20:42.3298528Z 
2022-11-23T02:20:42.3298589Z OK
2022-11-23T02:20:42.3298594Z 
2022-11-23T02:20:42.3298677Z Generating XML reports...
2022-11-23T02:20:42.3299022Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021208.xml
2022-11-23T02:20:42.3299300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3299412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3299694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3299833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3300021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3364cn9
2022-11-23T02:20:42.3300218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3364cn9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3300472Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3300477Z 
2022-11-23T02:20:42.3300548Z Running tests...
2022-11-23T02:20:42.3300740Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3300995Z   test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11010
2022-11-23T02:20:42.3301140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11011
2022-11-23T02:20:42.3301294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11012
2022-11-23T02:20:42.3301448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11013
2022-11-23T02:20:42.3301722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3301872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3302222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3302356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3302542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovkflu38
2022-11-23T02:20:42.3302727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovkflu38/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3303000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3303122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3303403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3303540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3303731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_wpgo0k
2022-11-23T02:20:42.3303926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_wpgo0k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3304200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3304323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3304585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3304718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3304901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8ky6hz5
2022-11-23T02:20:42.3305097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8ky6hz5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3305367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3305487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3305768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3305901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3306072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilqhtt90
2022-11-23T02:20:42.3306266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilqhtt90/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3306431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3306593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3306755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3306915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3306981Z ok (2.474s)
2022-11-23T02:20:42.3306986Z 
2022-11-23T02:20:42.3307182Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3307285Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3307291Z 
2022-11-23T02:20:42.3307338Z OK
2022-11-23T02:20:42.3307342Z 
2022-11-23T02:20:42.3307424Z Generating XML reports...
2022-11-23T02:20:42.3307763Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021212.xml
2022-11-23T02:20:42.3308035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3308158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3308464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3308600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3308781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ek45x8y
2022-11-23T02:20:42.3308963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ek45x8y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3309220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3309225Z 
2022-11-23T02:20:42.3309296Z Running tests...
2022-11-23T02:20:42.3309490Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3309703Z   test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11233
2022-11-23T02:20:42.3309857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11234
2022-11-23T02:20:42.3310010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11235
2022-11-23T02:20:42.3310162Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11236
2022-11-23T02:20:42.3310436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3310545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3310826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3310958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3311140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprn1euy9j
2022-11-23T02:20:42.3311334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprn1euy9j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3311604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3311726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3312005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3312140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3312309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ptsv2pc
2022-11-23T02:20:42.3312502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ptsv2pc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3312769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3312889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3313166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3313300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3313483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lnpvl9h
2022-11-23T02:20:42.3313675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lnpvl9h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3313959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3314080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3314358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3314488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3314670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmd5syl8v
2022-11-23T02:20:42.3314904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmd5syl8v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3315066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3315230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3315392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3315539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3315604Z ok (2.579s)
2022-11-23T02:20:42.3315610Z 
2022-11-23T02:20:42.3315806Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3315879Z Ran 1 test in 2.580s
2022-11-23T02:20:42.3315884Z 
2022-11-23T02:20:42.3315942Z OK
2022-11-23T02:20:42.3315946Z 
2022-11-23T02:20:42.3316026Z Generating XML reports...
2022-11-23T02:20:42.3316441Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021217.xml
2022-11-23T02:20:42.3316717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3316826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3317106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3317237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3317423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cv82qu0
2022-11-23T02:20:42.3317616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cv82qu0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3317869Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3317874Z 
2022-11-23T02:20:42.3317947Z Running tests...
2022-11-23T02:20:42.3318149Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3318370Z   test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11456
2022-11-23T02:20:42.3318514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11457
2022-11-23T02:20:42.3318667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11458
2022-11-23T02:20:42.3318817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11459
2022-11-23T02:20:42.3319088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3319210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3319492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3319627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3319811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27vi3q_r
2022-11-23T02:20:42.3319990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27vi3q_r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3320289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3320409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3320685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3320818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3321001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz98nsii
2022-11-23T02:20:42.3321267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3321412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3321684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3321805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3321986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxznontb
2022-11-23T02:20:42.3322177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz98nsii/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3322366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxznontb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3322634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3322755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3323035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3323166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3323336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrrldef1
2022-11-23T02:20:42.3323531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrrldef1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3323759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3323918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3324078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3324236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3324301Z ok (2.575s)
2022-11-23T02:20:42.3324308Z 
2022-11-23T02:20:42.3324505Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3324578Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3324583Z 
2022-11-23T02:20:42.3324630Z OK
2022-11-23T02:20:42.3324634Z 
2022-11-23T02:20:42.3324713Z Generating XML reports...
2022-11-23T02:20:42.3325153Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021221.xml
2022-11-23T02:20:42.3325433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3325555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3325833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3325965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3326145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfitsjshb
2022-11-23T02:20:42.3326327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfitsjshb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3326581Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3326586Z 
2022-11-23T02:20:42.3326657Z Running tests...
2022-11-23T02:20:42.3326890Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3327125Z   test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11679
2022-11-23T02:20:42.3327351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11680
2022-11-23T02:20:42.3327506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11681
2022-11-23T02:20:42.3327658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11682
2022-11-23T02:20:42.3327962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3328071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3328351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3328487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3328673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzlwel39
2022-11-23T02:20:42.3328869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzlwel39/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3329142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3329264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3329547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3329682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3329854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5xol64c
2022-11-23T02:20:42.3330113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5xol64c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3330385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3330505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3330784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3330918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3331100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6vv1fqa
2022-11-23T02:20:42.3331293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6vv1fqa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3331552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3331676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3331956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3332089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3332272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbofm4qze
2022-11-23T02:20:42.3332462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbofm4qze/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3332626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3332789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3332951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3333097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3333162Z ok (2.574s)
2022-11-23T02:20:42.3333197Z 
2022-11-23T02:20:42.3333395Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3333470Z Ran 1 test in 2.574s
2022-11-23T02:20:42.3333476Z 
2022-11-23T02:20:42.3333536Z OK
2022-11-23T02:20:42.3333541Z 
2022-11-23T02:20:42.3333622Z Generating XML reports...
2022-11-23T02:20:42.3333961Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021226.xml
2022-11-23T02:20:42.3334229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3334366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3334645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3334776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3335097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct0gzkcr
2022-11-23T02:20:42.3335296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct0gzkcr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3335557Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3335562Z 
2022-11-23T02:20:42.3335633Z Running tests...
2022-11-23T02:20:42.3335831Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3336062Z   test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11902
2022-11-23T02:20:42.3336206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11903
2022-11-23T02:20:42.3336361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11904
2022-11-23T02:20:42.3336510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11905
2022-11-23T02:20:42.3336783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3336906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3337186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3337319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3337502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpy07mj8
2022-11-23T02:20:42.3337687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpy07mj8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3337958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3338078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3338359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3338491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3338674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwmcath3
2022-11-23T02:20:42.3338869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwmcath3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3339134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3339255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3339523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3339654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3339897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6i9bkx9
2022-11-23T02:20:42.3340092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6i9bkx9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3340361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3340483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3340763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3340894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3341101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xi22l8p
2022-11-23T02:20:42.3341294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xi22l8p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3341456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3341619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3341780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3341939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3342005Z ok (2.677s)
2022-11-23T02:20:42.3342012Z 
2022-11-23T02:20:42.3342209Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3342283Z Ran 1 test in 2.678s
2022-11-23T02:20:42.3342288Z 
2022-11-23T02:20:42.3342338Z OK
2022-11-23T02:20:42.3342342Z 
2022-11-23T02:20:42.3342421Z Generating XML reports...
2022-11-23T02:20:42.3342760Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021230.xml
2022-11-23T02:20:42.3343032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3343154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3343521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3343658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3343840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42zosu42
2022-11-23T02:20:42.3344020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42zosu42/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3344271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3344278Z 
2022-11-23T02:20:42.3344350Z Running tests...
2022-11-23T02:20:42.3344546Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3344785Z   test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12125
2022-11-23T02:20:42.3344940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12126
2022-11-23T02:20:42.3345091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12127
2022-11-23T02:20:42.3345242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12128
2022-11-23T02:20:42.3345513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3345622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3345903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3346035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3346220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kfzg5b2
2022-11-23T02:20:42.3346440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kfzg5b2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3346707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3346827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3347103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3347234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3347406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpen0pwrip
2022-11-23T02:20:42.3347628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpen0pwrip/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3347895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3348022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3348298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3348432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3348612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkk7u3cbv
2022-11-23T02:20:42.3348871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkk7u3cbv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3349130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3349253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3349529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3349661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3349843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2oe62iet
2022-11-23T02:20:42.3350035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2oe62iet/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3350197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3350358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3350520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3350669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3350735Z ok (2.480s)
2022-11-23T02:20:42.3350740Z 
2022-11-23T02:20:42.3350935Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3351008Z Ran 1 test in 2.480s
2022-11-23T02:20:42.3351013Z 
2022-11-23T02:20:42.3351073Z OK
2022-11-23T02:20:42.3351080Z 
2022-11-23T02:20:42.3351162Z Generating XML reports...
2022-11-23T02:20:42.3351502Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021235.xml
2022-11-23T02:20:42.3351773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3351883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3352161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3352295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3352478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhxjvtm5
2022-11-23T02:20:42.3352672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhxjvtm5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3352955Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3352961Z 
2022-11-23T02:20:42.3353031Z Running tests...
2022-11-23T02:20:42.3353225Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3353456Z   test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12348
2022-11-23T02:20:42.3353600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12349
2022-11-23T02:20:42.3353751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12350
2022-11-23T02:20:42.3353929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12351
2022-11-23T02:20:42.3354203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3354325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3354604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3354735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3354918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgs3yqj9
2022-11-23T02:20:42.3355101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgs3yqj9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3355371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3355493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3355771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3355904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3356088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nnapv05
2022-11-23T02:20:42.3356281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nnapv05/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3356546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3356665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3356931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3357067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3357248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4dzwdam
2022-11-23T02:20:42.3357439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4dzwdam/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3357710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3357831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3358107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3358240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3358409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18txd8i0
2022-11-23T02:20:42.3358599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18txd8i0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3358762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3358925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3359085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3359273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3359339Z ok (2.575s)
2022-11-23T02:20:42.3359344Z 
2022-11-23T02:20:42.3359539Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3359611Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3359616Z 
2022-11-23T02:20:42.3359662Z OK
2022-11-23T02:20:42.3359667Z 
2022-11-23T02:20:42.3359746Z Generating XML reports...
2022-11-23T02:20:42.3360083Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021239.xml
2022-11-23T02:20:42.3360394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3360515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3360792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3361062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3361249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya5xat3l
2022-11-23T02:20:42.3361428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya5xat3l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3361685Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3361690Z 
2022-11-23T02:20:42.3361762Z Running tests...
2022-11-23T02:20:42.3361958Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3362175Z   test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12571
2022-11-23T02:20:42.3362334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12572
2022-11-23T02:20:42.3362488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12573
2022-11-23T02:20:42.3362642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12574
2022-11-23T02:20:42.3362913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3363021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3363299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3363430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3363616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xw1tshv
2022-11-23T02:20:42.3363811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xw1tshv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3364080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3364205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3364484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3364616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3364786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xbaty8f
2022-11-23T02:20:42.3365054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xbaty8f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3365325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3365449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3365727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3365894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3366079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi64hxm9
2022-11-23T02:20:42.3366270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi64hxm9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3366524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3366645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3366924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3367115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3367295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttricenz
2022-11-23T02:20:42.3367486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttricenz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3367650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3367811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3367973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3368122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3368187Z ok (2.480s)
2022-11-23T02:20:42.3368192Z 
2022-11-23T02:20:42.3368462Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3368536Z Ran 1 test in 2.480s
2022-11-23T02:20:42.3368541Z 
2022-11-23T02:20:42.3368602Z OK
2022-11-23T02:20:42.3368607Z 
2022-11-23T02:20:42.3368688Z Generating XML reports...
2022-11-23T02:20:42.3369024Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021243.xml
2022-11-23T02:20:42.3369295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3369405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3369682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3369815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3369999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec1vo5s4
2022-11-23T02:20:42.3370192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec1vo5s4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3370445Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3370449Z 
2022-11-23T02:20:42.3370521Z Running tests...
2022-11-23T02:20:42.3370719Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3370956Z   test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12794
2022-11-23T02:20:42.3371100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12795
2022-11-23T02:20:42.3371253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12796
2022-11-23T02:20:42.3371401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12797
2022-11-23T02:20:42.3371671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3371794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3372142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3372276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3372485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp092yfjuv
2022-11-23T02:20:42.3372666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp092yfjuv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3372940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3373060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3373338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3373500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3373683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hd2javf
2022-11-23T02:20:42.3373877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hd2javf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3374147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3374266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3374534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3374667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3374948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv846md5d
2022-11-23T02:20:42.3375178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv846md5d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3375455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3375577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3375858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3375992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3376162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq41haqfc
2022-11-23T02:20:42.3376355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq41haqfc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3376517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3376680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3376846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3377004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3377071Z ok (2.577s)
2022-11-23T02:20:42.3377077Z 
2022-11-23T02:20:42.3377276Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3377350Z Ran 1 test in 2.578s
2022-11-23T02:20:42.3377355Z 
2022-11-23T02:20:42.3377402Z OK
2022-11-23T02:20:42.3377406Z 
2022-11-23T02:20:42.3377487Z Generating XML reports...
2022-11-23T02:20:42.3377822Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021248.xml
2022-11-23T02:20:42.3378094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3378214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3378492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3378627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3378808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu0oizx9
2022-11-23T02:20:42.3379050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu0oizx9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3379305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3379310Z 
2022-11-23T02:20:42.3379380Z Running tests...
2022-11-23T02:20:42.3379574Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3379810Z   test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13017
2022-11-23T02:20:42.3379967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13018
2022-11-23T02:20:42.3380158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13019
2022-11-23T02:20:42.3380311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13020
2022-11-23T02:20:42.3380585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3380694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3380977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3381108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3381291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv_w60bt
2022-11-23T02:20:42.3381482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv_w60bt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3381752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3381872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3382150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3382283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3382454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0tgx6f6
2022-11-23T02:20:42.3382644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0tgx6f6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3382912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3383035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3383311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3383448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3383631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2npytvj
2022-11-23T02:20:42.3383821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2npytvj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3384078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3384198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3384606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3384737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3384920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j7flc_o
2022-11-23T02:20:42.3385115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j7flc_o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3385277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3385437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3385627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3385777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3385842Z ok (2.473s)
2022-11-23T02:20:42.3385847Z 
2022-11-23T02:20:42.3386050Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3386125Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3386130Z 
2022-11-23T02:20:42.3386188Z OK
2022-11-23T02:20:42.3386192Z 
2022-11-23T02:20:42.3386274Z Generating XML reports...
2022-11-23T02:20:42.3386640Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021252.xml
2022-11-23T02:20:42.3386910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3387019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3387295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3387429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3387611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8_hlny6
2022-11-23T02:20:42.3387802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8_hlny6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3388053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3388060Z 
2022-11-23T02:20:42.3388131Z Running tests...
2022-11-23T02:20:42.3388324Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3388539Z   test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13240
2022-11-23T02:20:42.3388751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13241
2022-11-23T02:20:42.3388901Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13242
2022-11-23T02:20:42.3389048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13243
2022-11-23T02:20:42.3389322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3389444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3389723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3389857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3390039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tj2m6wa
2022-11-23T02:20:42.3390219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tj2m6wa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3390491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3390611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3390894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3391030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3391212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0ljl__8
2022-11-23T02:20:42.3391404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0ljl__8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3391675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3391801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3392097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3392231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3392412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp76d89in
2022-11-23T02:20:42.3392602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp76d89in/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3392868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3392988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3393293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3393426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3393666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkee3jjrm
2022-11-23T02:20:42.3393858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkee3jjrm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3394017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3394178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3394337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3394495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3394674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3394848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3395151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3395312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3395479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3395773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3396071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3396360Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3396628Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3396912Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down
2022-11-23T02:20:42.3397165Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3397430Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3397677Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3397938Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3398225Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3398475Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3399013Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker3', 'worker1', 'worker2'] timed out in _all_gather after 0.00 seconds. The first exception is eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3399139Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3399255Z RuntimeError('simulation')
2022-11-23T02:20:42.3399344Z Traceback (most recent call last):
2022-11-23T02:20:42.3399606Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3399740Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3400024Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2022-11-23T02:20:42.3400156Z     raise RuntimeError('simulation')
2022-11-23T02:20:42.3400226Z RuntimeError: simulation
2022-11-23T02:20:42.3400231Z 
2022-11-23T02:20:42.3400320Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3400434Z RuntimeError('simulation')
2022-11-23T02:20:42.3400521Z Traceback (most recent call last):
2022-11-23T02:20:42.3400778Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3400911Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3401197Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2022-11-23T02:20:42.3401325Z     raise RuntimeError('simulation')
2022-11-23T02:20:42.3401396Z RuntimeError: simulation
2022-11-23T02:20:42.3401401Z 
2022-11-23T02:20:42.3401490Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3401603Z RuntimeError('simulation')
2022-11-23T02:20:42.3401692Z Traceback (most recent call last):
2022-11-23T02:20:42.3402109Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3402240Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3402521Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4561, in raise_error
2022-11-23T02:20:42.3402643Z     raise RuntimeError('simulation')
2022-11-23T02:20:42.3402727Z RuntimeError: simulation
2022-11-23T02:20:42.3402732Z 
2022-11-23T02:20:42.3402799Z ok (2.576s)
2022-11-23T02:20:42.3402804Z 
2022-11-23T02:20:42.3403001Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3403074Z Ran 1 test in 2.576s
2022-11-23T02:20:42.3403081Z 
2022-11-23T02:20:42.3403141Z OK
2022-11-23T02:20:42.3403146Z 
2022-11-23T02:20:42.3403227Z Generating XML reports...
2022-11-23T02:20:42.3403567Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021257.xml
2022-11-23T02:20:42.3403844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3403956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3404242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3404379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3404564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzp7qu4p
2022-11-23T02:20:42.3404794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzp7qu4p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3405114Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3405120Z 
2022-11-23T02:20:42.3405193Z Running tests...
2022-11-23T02:20:42.3405389Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3405606Z   test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13475
2022-11-23T02:20:42.3405761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13476
2022-11-23T02:20:42.3405962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13477
2022-11-23T02:20:42.3406115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13478
2022-11-23T02:20:42.3406394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3406518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3406800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3406934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3407117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxsydn4l
2022-11-23T02:20:42.3407299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxsydn4l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3407570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3407696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3407979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3408108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3408295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswi_5llo
2022-11-23T02:20:42.3408484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswi_5llo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3408754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3408862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3409140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3409276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3409458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86_bp72m
2022-11-23T02:20:42.3409648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86_bp72m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3409919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3410041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3410318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3410448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3410618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf0ppspy
2022-11-23T02:20:42.3410816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf0ppspy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3410977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3411140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3411332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3411492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3411557Z ok (2.578s)
2022-11-23T02:20:42.3411563Z 
2022-11-23T02:20:42.3411758Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3411820Z Ran 1 test in 2.579s
2022-11-23T02:20:42.3411836Z 
2022-11-23T02:20:42.3411882Z OK
2022-11-23T02:20:42.3411886Z 
2022-11-23T02:20:42.3411968Z Generating XML reports...
2022-11-23T02:20:42.3412305Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021301.xml
2022-11-23T02:20:42.3412608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3412729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3413007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3413139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3413319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_pokf18
2022-11-23T02:20:42.3413499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_pokf18/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3413863Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3413869Z 
2022-11-23T02:20:42.3413942Z Running tests...
2022-11-23T02:20:42.3414143Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3414365Z   test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13698
2022-11-23T02:20:42.3414522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13699
2022-11-23T02:20:42.3414677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13700
2022-11-23T02:20:42.3414828Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13701
2022-11-23T02:20:42.3415254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3415367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3415652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3415789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3415976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswksnebp
2022-11-23T02:20:42.3416174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswksnebp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3416449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3416570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3416851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3416974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3417159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigozjali
2022-11-23T02:20:42.3417356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigozjali/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3417630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3417752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3418027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3418217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3418404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac6788xx
2022-11-23T02:20:42.3418599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac6788xx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3418859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3418983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3419298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3419434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3419620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh0lp14c
2022-11-23T02:20:42.3419816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh0lp14c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3419980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3420143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3420291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3420634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3420752Z Rank 2 finished testing 20 times in 0.07480525970458984 seconds.
2022-11-23T02:20:42.3420869Z Rank 3 finished testing 20 times in 0.0710608959197998 seconds.
2022-11-23T02:20:42.3420982Z Rank 0 finished testing 20 times in 0.0822756290435791 seconds.
2022-11-23T02:20:42.3421096Z Rank 1 finished testing 20 times in 0.08189797401428223 seconds.
2022-11-23T02:20:42.3421161Z ok (2.575s)
2022-11-23T02:20:42.3421166Z 
2022-11-23T02:20:42.3421369Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3421429Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3421446Z 
2022-11-23T02:20:42.3421492Z OK
2022-11-23T02:20:42.3421497Z 
2022-11-23T02:20:42.3421580Z Generating XML reports...
2022-11-23T02:20:42.3421923Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021305.xml
2022-11-23T02:20:42.3422199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3422321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3422606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3422740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3422923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33i3sj24
2022-11-23T02:20:42.3423105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33i3sj24/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3423359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3423363Z 
2022-11-23T02:20:42.3423435Z Running tests...
2022-11-23T02:20:42.3423697Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3423930Z   test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13921
2022-11-23T02:20:42.3424087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13922
2022-11-23T02:20:42.3424237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13923
2022-11-23T02:20:42.3424385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13924
2022-11-23T02:20:42.3424681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3424800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3425083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3425219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3425403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr7ht74w
2022-11-23T02:20:42.3425599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr7ht74w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3425900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3426023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3426305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3426425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3426606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__qgyu_v
2022-11-23T02:20:42.3426796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__qgyu_v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3427063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3427180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3427458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3427590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3427772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf64tbcz
2022-11-23T02:20:42.3427965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf64tbcz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3428224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3428342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3428614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3428744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3428990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_8_wicl
2022-11-23T02:20:42.3429181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_8_wicl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3429342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3429506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3429655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3429814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3429926Z Rank 2 finished testing 20 times in 0.10598397254943848 seconds.
2022-11-23T02:20:42.3430039Z Rank 3 finished testing 20 times in 0.0916757583618164 seconds.
2022-11-23T02:20:42.3430150Z Rank 0 finished testing 20 times in 0.11051344871520996 seconds.
2022-11-23T02:20:42.3430258Z Rank 1 finished testing 20 times in 0.11762881278991699 seconds.
2022-11-23T02:20:42.3430327Z ok (2.577s)
2022-11-23T02:20:42.3430333Z 
2022-11-23T02:20:42.3430531Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3430592Z Ran 1 test in 2.577s
2022-11-23T02:20:42.3430597Z 
2022-11-23T02:20:42.3430655Z OK
2022-11-23T02:20:42.3430660Z 
2022-11-23T02:20:42.3430744Z Generating XML reports...
2022-11-23T02:20:42.3431119Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021310.xml
2022-11-23T02:20:42.3431394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3431512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3431786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3431916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3432129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgoah7c7m
2022-11-23T02:20:42.3432311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgoah7c7m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3432566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3432573Z 
2022-11-23T02:20:42.3432642Z Running tests...
2022-11-23T02:20:42.3432836Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3433054Z   test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14144
2022-11-23T02:20:42.3433206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14145
2022-11-23T02:20:42.3433357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14146
2022-11-23T02:20:42.3433508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14147
2022-11-23T02:20:42.3433771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3433890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3434169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3434302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3434487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc1v5o0h
2022-11-23T02:20:42.3434680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc1v5o0h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3434950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3435068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3435348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3435469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3435649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mxpaumv
2022-11-23T02:20:42.3435955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mxpaumv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3436228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3436347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3436622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3436755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3436942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjax99lus
2022-11-23T02:20:42.3437122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjax99lus/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3437391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3437543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3437823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3437955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3438135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkakigtww
2022-11-23T02:20:42.3438327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkakigtww/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3438490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3438684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3438831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3438990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3439103Z Rank 0 finished testing 1000 times in 0.2900214195251465 seconds.
2022-11-23T02:20:42.3439213Z Rank 2 finished testing 1000 times in 0.3081808090209961 seconds.
2022-11-23T02:20:42.3439325Z Rank 3 finished testing 1000 times in 0.282153844833374 seconds.
2022-11-23T02:20:42.3439439Z Rank 1 finished testing 1000 times in 0.28249073028564453 seconds.
2022-11-23T02:20:42.3439503Z ok (2.778s)
2022-11-23T02:20:42.3439509Z 
2022-11-23T02:20:42.3439702Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3439763Z Ran 1 test in 2.778s
2022-11-23T02:20:42.3439770Z 
2022-11-23T02:20:42.3439828Z OK
2022-11-23T02:20:42.3439833Z 
2022-11-23T02:20:42.3439914Z Generating XML reports...
2022-11-23T02:20:42.3440253Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021314.xml
2022-11-23T02:20:42.3440529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3440651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3440934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3441066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3441238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporp87cs3
2022-11-23T02:20:42.3441433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporp87cs3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3441687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3441692Z 
2022-11-23T02:20:42.3441762Z Running tests...
2022-11-23T02:20:42.3441956Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3442178Z   test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14367
2022-11-23T02:20:42.3442333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14368
2022-11-23T02:20:42.3442485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14369
2022-11-23T02:20:42.3442636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14370
2022-11-23T02:20:42.3442896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3443017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3443298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3443594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3443782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt07mphvc
2022-11-23T02:20:42.3444010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt07mphvc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3444284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3444407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3444685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3444804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3445082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fl2cu01
2022-11-23T02:20:42.3445274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fl2cu01/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3445550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3445674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3445954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3446086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3446268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3dklqqj
2022-11-23T02:20:42.3446448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3dklqqj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3446790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3446909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3447190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3447322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3447501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vqk3m8k
2022-11-23T02:20:42.3447692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vqk3m8k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3447855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3448018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3448167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3448329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3448396Z ok (2.173s)
2022-11-23T02:20:42.3448401Z 
2022-11-23T02:20:42.3448598Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3448672Z Ran 1 test in 2.173s
2022-11-23T02:20:42.3448677Z 
2022-11-23T02:20:42.3448740Z OK
2022-11-23T02:20:42.3448745Z 
2022-11-23T02:20:42.3448826Z Generating XML reports...
2022-11-23T02:20:42.3449165Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021319.xml
2022-11-23T02:20:42.3449426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3449548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3449828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3449963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3450209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjnx68lg
2022-11-23T02:20:42.3450404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjnx68lg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3450689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3450695Z 
2022-11-23T02:20:42.3450765Z Running tests...
2022-11-23T02:20:42.3450957Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3451171Z   test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14422
2022-11-23T02:20:42.3451324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14423
2022-11-23T02:20:42.3451475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14424
2022-11-23T02:20:42.3451671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14425
2022-11-23T02:20:42.3451943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3452063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3452348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3452483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3452653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87t54l2o
2022-11-23T02:20:42.3452845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87t54l2o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3453115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3453240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3453517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3453651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3453836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdieh92s
2022-11-23T02:20:42.3454029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdieh92s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3454297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3454407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3454682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3454817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3455152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd3e7ki1
2022-11-23T02:20:42.3455351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd3e7ki1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3455628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3455750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3456027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3456149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3456333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3rt4ck9
2022-11-23T02:20:42.3456527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3rt4ck9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3456693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3456854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3457074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3457238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3457306Z ok (2.575s)
2022-11-23T02:20:42.3457311Z 
2022-11-23T02:20:42.3457507Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3457569Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3457574Z 
2022-11-23T02:20:42.3457634Z OK
2022-11-23T02:20:42.3457639Z 
2022-11-23T02:20:42.3457722Z Generating XML reports...
2022-11-23T02:20:42.3458067Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021323.xml
2022-11-23T02:20:42.3458377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3458500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3458782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3458915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3459086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25rh04ad
2022-11-23T02:20:42.3459464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25rh04ad/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3459729Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3459734Z 
2022-11-23T02:20:42.3459805Z Running tests...
2022-11-23T02:20:42.3460002Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3460223Z   test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14645
2022-11-23T02:20:42.3460366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14646
2022-11-23T02:20:42.3460522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14647
2022-11-23T02:20:42.3460670Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14648
2022-11-23T02:20:42.3460943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3461063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3461345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3461478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3461663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a499vgp
2022-11-23T02:20:42.3461844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a499vgp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3462113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3462233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3462511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3462645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3462828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsjpbpuq
2022-11-23T02:20:42.3463023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsjpbpuq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3463293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3463412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3463747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3463912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3464097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwztimyti
2022-11-23T02:20:42.3464292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwztimyti/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3464564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3464684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3464962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3465134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3465313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd2ggkul
2022-11-23T02:20:42.3465497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd2ggkul/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3465659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3465820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3465983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3466141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3466206Z ok (2.674s)
2022-11-23T02:20:42.3466212Z 
2022-11-23T02:20:42.3466407Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3466477Z Ran 1 test in 2.674s
2022-11-23T02:20:42.3466482Z 
2022-11-23T02:20:42.3466528Z OK
2022-11-23T02:20:42.3466533Z 
2022-11-23T02:20:42.3466614Z Generating XML reports...
2022-11-23T02:20:42.3466952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021327.xml
2022-11-23T02:20:42.3467224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3467345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3467620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3467752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3467933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0eae0oo
2022-11-23T02:20:42.3468116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0eae0oo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3468438Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3468443Z 
2022-11-23T02:20:42.3468514Z Running tests...
2022-11-23T02:20:42.3468710Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3468930Z   test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14888
2022-11-23T02:20:42.3469084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14889
2022-11-23T02:20:42.3469235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14890
2022-11-23T02:20:42.3469384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14891
2022-11-23T02:20:42.3469657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3469768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3470044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3470177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3470428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dojhw3x
2022-11-23T02:20:42.3470621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dojhw3x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3470890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3471012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3471289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3471453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3471625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk213an_l
2022-11-23T02:20:42.3471815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk213an_l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3472087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3472208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3472483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3472615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3472794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5_vviik
2022-11-23T02:20:42.3472981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5_vviik/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3473238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3473358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3473634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3473765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3473946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtn3di3h
2022-11-23T02:20:42.3474138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtn3di3h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3474298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3474456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3474617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3474762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3474827Z ok (2.673s)
2022-11-23T02:20:42.3474832Z 
2022-11-23T02:20:42.3475029Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3475102Z Ran 1 test in 2.673s
2022-11-23T02:20:42.3475106Z 
2022-11-23T02:20:42.3475164Z OK
2022-11-23T02:20:42.3475169Z 
2022-11-23T02:20:42.3475248Z Generating XML reports...
2022-11-23T02:20:42.3475584Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021332.xml
2022-11-23T02:20:42.3475855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3475964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3476242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3476375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3476561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldrmp3se
2022-11-23T02:20:42.3476783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldrmp3se/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3477035Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3477040Z 
2022-11-23T02:20:42.3477109Z Running tests...
2022-11-23T02:20:42.3477302Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3477533Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15111
2022-11-23T02:20:42.3477704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15112
2022-11-23T02:20:42.3477856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15113
2022-11-23T02:20:42.3478004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15114
2022-11-23T02:20:42.3478283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3478404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3478686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3478819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3479002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5fu6yix
2022-11-23T02:20:42.3479304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5fu6yix/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3479575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3479697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3479976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3480111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3480295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pr6isap
2022-11-23T02:20:42.3480508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pr6isap/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3480778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3480899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3481166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3481297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3481480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1o3ozo2
2022-11-23T02:20:42.3481672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1o3ozo2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3481939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3482060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3482336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3482468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3482649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxf5fr8k
2022-11-23T02:20:42.3482832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxf5fr8k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3482992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3483185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3483346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3483505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3483569Z ok (2.674s)
2022-11-23T02:20:42.3483575Z 
2022-11-23T02:20:42.3483768Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3483842Z Ran 1 test in 2.675s
2022-11-23T02:20:42.3483847Z 
2022-11-23T02:20:42.3483893Z OK
2022-11-23T02:20:42.3483898Z 
2022-11-23T02:20:42.3483977Z Generating XML reports...
2022-11-23T02:20:42.3484340Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021337.xml
2022-11-23T02:20:42.3484610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3484800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3485152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3485288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3485469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_09gtiv1
2022-11-23T02:20:42.3485661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_09gtiv1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3485903Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3485910Z 
2022-11-23T02:20:42.3485980Z Running tests...
2022-11-23T02:20:42.3486169Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3486376Z   test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15334
2022-11-23T02:20:42.3486531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15335
2022-11-23T02:20:42.3486679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15336
2022-11-23T02:20:42.3486829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15337
2022-11-23T02:20:42.3487100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3487208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3487485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3487618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3487802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczpch__i
2022-11-23T02:20:42.3487995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczpch__i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3488267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3488386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3488728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3488858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3489027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_ohclgm
2022-11-23T02:20:42.3489222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_ohclgm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3489493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3489616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3489928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3490061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3490241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_k_l041
2022-11-23T02:20:42.3490428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_k_l041/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3490686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3490832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3491106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3491239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3491423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv06uenzg
2022-11-23T02:20:42.3491615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv06uenzg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3491775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3491937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3492095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3492244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3492309Z ok (2.476s)
2022-11-23T02:20:42.3492314Z 
2022-11-23T02:20:42.3492509Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3492581Z Ran 1 test in 2.477s
2022-11-23T02:20:42.3492586Z 
2022-11-23T02:20:42.3492644Z OK
2022-11-23T02:20:42.3492649Z 
2022-11-23T02:20:42.3492729Z Generating XML reports...
2022-11-23T02:20:42.3493066Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021341.xml
2022-11-23T02:20:42.3493337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3493512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3493792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3493924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3494107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eu_as2p
2022-11-23T02:20:42.3494300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eu_as2p/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3494554Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3494559Z 
2022-11-23T02:20:42.3494629Z Running tests...
2022-11-23T02:20:42.3494820Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3495138Z   test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15557
2022-11-23T02:20:42.3495281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15558
2022-11-23T02:20:42.3495433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15559
2022-11-23T02:20:42.3495582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15560
2022-11-23T02:20:42.3495859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3495979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3496311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3496447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3496629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm22f8cee
2022-11-23T02:20:42.3496809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm22f8cee/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3497078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3497198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3497523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3497654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3497838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4ajkw1j
2022-11-23T02:20:42.3498031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4ajkw1j/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3498298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3498417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3498681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3498812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3498993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xlgg_bu
2022-11-23T02:20:42.3499184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xlgg_bu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3499455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3499576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3499856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3499989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3500158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsyb28xs
2022-11-23T02:20:42.3500349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsyb28xs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3500510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3500673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3500832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3500989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3501165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3501337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3501507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3501666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3501965Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3502265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3502556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3502874Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3502939Z ok (2.686s)
2022-11-23T02:20:42.3502945Z 
2022-11-23T02:20:42.3503134Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3503207Z Ran 1 test in 2.686s
2022-11-23T02:20:42.3503212Z 
2022-11-23T02:20:42.3503270Z OK
2022-11-23T02:20:42.3503274Z 
2022-11-23T02:20:42.3503344Z Generating XML reports...
2022-11-23T02:20:42.3503677Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021346.xml
2022-11-23T02:20:42.3503977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3504100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3504380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3504513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3504768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumu4jp3t
2022-11-23T02:20:42.3504961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumu4jp3t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3505214Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3505219Z 
2022-11-23T02:20:42.3505279Z Running tests...
2022-11-23T02:20:42.3505473Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3505702Z   test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15792
2022-11-23T02:20:42.3505853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15793
2022-11-23T02:20:42.3506006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15794
2022-11-23T02:20:42.3506154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15795
2022-11-23T02:20:42.3506426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3506544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3506813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3506945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3507128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w_wlf8b
2022-11-23T02:20:42.3507318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w_wlf8b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3507587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3507710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3507985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3508115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3508294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoffhdcq
2022-11-23T02:20:42.3508475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoffhdcq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3508743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3508862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3509138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3509311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3509493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphplzr6e1
2022-11-23T02:20:42.3509687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphplzr6e1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3509955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3510074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3510337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3510496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3510678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaor73bc6
2022-11-23T02:20:42.3510871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaor73bc6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3511032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3511196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3511356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3511516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3511677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3511853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3512024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3512321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3512616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3512788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3513078Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3513368Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3513460Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3513566Z ValueError('Expected error')
2022-11-23T02:20:42.3513655Z Traceback (most recent call last):
2022-11-23T02:20:42.3513915Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3514048Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3514328Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3514415Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3514495Z ValueError: Expected error
2022-11-23T02:20:42.3514501Z 
2022-11-23T02:20:42.3514588Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3514693Z ValueError('Expected error')
2022-11-23T02:20:42.3514781Z Traceback (most recent call last):
2022-11-23T02:20:42.3515038Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3515172Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3515446Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3515536Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3515626Z ValueError: Expected error
2022-11-23T02:20:42.3515631Z 
2022-11-23T02:20:42.3515744Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3515850Z ValueError('Expected error')
2022-11-23T02:20:42.3515937Z Traceback (most recent call last):
2022-11-23T02:20:42.3516310Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3516441Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3516719Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3516806Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3516922Z ValueError: Expected error
2022-11-23T02:20:42.3516927Z 
2022-11-23T02:20:42.3517001Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3517117Z ValueError('Expected error')
2022-11-23T02:20:42.3517202Z Traceback (most recent call last):
2022-11-23T02:20:42.3517454Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3517580Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3517853Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3517941Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3518022Z ValueError: Expected error
2022-11-23T02:20:42.3518028Z 
2022-11-23T02:20:42.3518103Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3518216Z ValueError('Expected error')
2022-11-23T02:20:42.3518303Z Traceback (most recent call last):
2022-11-23T02:20:42.3518550Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3518677Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3518949Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3519042Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3519124Z ValueError: Expected error
2022-11-23T02:20:42.3519128Z 
2022-11-23T02:20:42.3519203Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3519318Z ValueError('Expected error')
2022-11-23T02:20:42.3519406Z Traceback (most recent call last):
2022-11-23T02:20:42.3519654Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3519779Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3520049Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3520140Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3520221Z ValueError: Expected error
2022-11-23T02:20:42.3520225Z 
2022-11-23T02:20:42.3520300Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3520414Z ValueError('Expected error')
2022-11-23T02:20:42.3520503Z Traceback (most recent call last):
2022-11-23T02:20:42.3520752Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3520877Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3521146Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3521233Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3521304Z ValueError: Expected error
2022-11-23T02:20:42.3521319Z 
2022-11-23T02:20:42.3521393Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3521510Z ValueError('Expected error')
2022-11-23T02:20:42.3521597Z Traceback (most recent call last):
2022-11-23T02:20:42.3521845Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3521972Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3522265Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3522355Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3522425Z ValueError: Expected error
2022-11-23T02:20:42.3522430Z 
2022-11-23T02:20:42.3522493Z ok (2.679s)
2022-11-23T02:20:42.3522498Z 
2022-11-23T02:20:42.3522691Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3522767Z Ran 1 test in 2.680s
2022-11-23T02:20:42.3522772Z 
2022-11-23T02:20:42.3522829Z OK
2022-11-23T02:20:42.3522834Z 
2022-11-23T02:20:42.3522913Z Generating XML reports...
2022-11-23T02:20:42.3523278Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021350.xml
2022-11-23T02:20:42.3523549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3523672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3523940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3524073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3524254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8srlb786
2022-11-23T02:20:42.3524445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8srlb786/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3524694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3524701Z 
2022-11-23T02:20:42.3524768Z Running tests...
2022-11-23T02:20:42.3525052Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3525292Z   test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16027
2022-11-23T02:20:42.3525438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16028
2022-11-23T02:20:42.3525593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16029
2022-11-23T02:20:42.3525742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16030
2022-11-23T02:20:42.3526019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3526139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3526420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3526553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3526737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0tiq1c_
2022-11-23T02:20:42.3526931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0tiq1c_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3527189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3527426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3527711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3527843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3528027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9egwqyzy
2022-11-23T02:20:42.3528223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9egwqyzy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3528491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3528611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3528926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3529046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3529227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafwg0pmx
2022-11-23T02:20:42.3529421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafwg0pmx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3529688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3529836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3530110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3530239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3530421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb37x_k0l
2022-11-23T02:20:42.3530598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb37x_k0l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3530758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3530919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3531077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3531234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3531409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3531739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3531912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3532084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3532373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3532669Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3532955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3533246Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3533311Z ok (2.676s)
2022-11-23T02:20:42.3533316Z 
2022-11-23T02:20:42.3533510Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3533581Z Ran 1 test in 2.676s
2022-11-23T02:20:42.3533586Z 
2022-11-23T02:20:42.3533646Z OK
2022-11-23T02:20:42.3533651Z 
2022-11-23T02:20:42.3533732Z Generating XML reports...
2022-11-23T02:20:42.3534058Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021355.xml
2022-11-23T02:20:42.3534329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3534448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3534727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3535043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3535253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwwty07y
2022-11-23T02:20:42.3535447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwwty07y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3535771Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3535777Z 
2022-11-23T02:20:42.3535836Z Running tests...
2022-11-23T02:20:42.3536032Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3536256Z   test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16262
2022-11-23T02:20:42.3536411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16263
2022-11-23T02:20:42.3536565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16264
2022-11-23T02:20:42.3536753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16265
2022-11-23T02:20:42.3537025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3537143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3537426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3537550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3537735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnii_6sg
2022-11-23T02:20:42.3537928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnii_6sg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3538199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3538326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3538604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3538735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3538920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw00ihof7
2022-11-23T02:20:42.3539102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw00ihof7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3539371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3539491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3539768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3539974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3540156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchwkvdjr
2022-11-23T02:20:42.3540350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchwkvdjr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3540622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3540742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3541012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3541146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3541326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph02wsmrv
2022-11-23T02:20:42.3541514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph02wsmrv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3541676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3541835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3541996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3542183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3542237Z ok (2.579s)
2022-11-23T02:20:42.3542254Z 
2022-11-23T02:20:42.3542438Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3542509Z Ran 1 test in 2.579s
2022-11-23T02:20:42.3542514Z 
2022-11-23T02:20:42.3542573Z OK
2022-11-23T02:20:42.3542578Z 
2022-11-23T02:20:42.3542659Z Generating XML reports...
2022-11-23T02:20:42.3543000Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021359.xml
2022-11-23T02:20:42.3543309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3543431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3543710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3543832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3544015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1vavi30
2022-11-23T02:20:42.3544206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1vavi30/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3544460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3544465Z 
2022-11-23T02:20:42.3544536Z Running tests...
2022-11-23T02:20:42.3544729Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3544954Z   test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16485
2022-11-23T02:20:42.3545108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16486
2022-11-23T02:20:42.3545261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16487
2022-11-23T02:20:42.3545401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16488
2022-11-23T02:20:42.3545673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3545793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3546073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3546204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3546386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r66xq6i
2022-11-23T02:20:42.3546653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3546773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3547041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3547173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3547353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3guvo05z
2022-11-23T02:20:42.3547543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r66xq6i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3547732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3guvo05z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3548004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3548126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3548403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3548564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3548735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cty97ke
2022-11-23T02:20:42.3548925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cty97ke/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3549194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3549313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3549587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3549934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3550115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4joyugt4
2022-11-23T02:20:42.3550307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4joyugt4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3550457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3550615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3550774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3550934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3551001Z ok (2.476s)
2022-11-23T02:20:42.3551006Z 
2022-11-23T02:20:42.3551206Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3551276Z Ran 1 test in 2.476s
2022-11-23T02:20:42.3551281Z 
2022-11-23T02:20:42.3551338Z OK
2022-11-23T02:20:42.3551343Z 
2022-11-23T02:20:42.3551423Z Generating XML reports...
2022-11-23T02:20:42.3551748Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021404.xml
2022-11-23T02:20:42.3552025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3552146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3552422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3552554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3552736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2gtm47h
2022-11-23T02:20:42.3552931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2gtm47h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3553182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3553188Z 
2022-11-23T02:20:42.3553247Z Running tests...
2022-11-23T02:20:42.3553444Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3553735Z   test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16708
2022-11-23T02:20:42.3553889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16709
2022-11-23T02:20:42.3554042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16710
2022-11-23T02:20:42.3554192Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16711
2022-11-23T02:20:42.3554464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3554585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3554861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3554982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3555196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte56wpxt
2022-11-23T02:20:42.3555387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte56wpxt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3555657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3555778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3556055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3556214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3556393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv52uh0q
2022-11-23T02:20:42.3556573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv52uh0q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3556844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3556964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3557243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3557439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3557618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl845ruf
2022-11-23T02:20:42.3557807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl845ruf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3558074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3558192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3558458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3558589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3558768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_4x_5sf
2022-11-23T02:20:42.3558955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_4x_5sf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3559114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3559275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3559433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3559590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3559669Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3559787Z ValueError('Expected error')
2022-11-23T02:20:42.3559878Z Traceback (most recent call last):
2022-11-23T02:20:42.3560140Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3560271Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3560549Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3560639Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3560720Z ValueError: Expected error
2022-11-23T02:20:42.3560725Z 
2022-11-23T02:20:42.3560803Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3560920Z ValueError('Expected error')
2022-11-23T02:20:42.3561007Z Traceback (most recent call last):
2022-11-23T02:20:42.3561262Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3561392Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3561701Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3561789Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3561870Z ValueError: Expected error
2022-11-23T02:20:42.3561875Z 
2022-11-23T02:20:42.3561950Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3562069Z ValueError('Expected error')
2022-11-23T02:20:42.3562158Z Traceback (most recent call last):
2022-11-23T02:20:42.3562411Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3562568Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3562844Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3562932Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3563012Z ValueError: Expected error
2022-11-23T02:20:42.3563017Z 
2022-11-23T02:20:42.3563094Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3563206Z ValueError('Expected error')
2022-11-23T02:20:42.3563292Z Traceback (most recent call last):
2022-11-23T02:20:42.3563542Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3563669Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3563942Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3564033Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3564103Z ValueError: Expected error
2022-11-23T02:20:42.3564117Z 
2022-11-23T02:20:42.3564170Z ok (2.474s)
2022-11-23T02:20:42.3564175Z 
2022-11-23T02:20:42.3564370Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3564441Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3564446Z 
2022-11-23T02:20:42.3564504Z OK
2022-11-23T02:20:42.3564511Z 
2022-11-23T02:20:42.3564593Z Generating XML reports...
2022-11-23T02:20:42.3565109Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021408.xml
2022-11-23T02:20:42.3565389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3565512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3565780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3565915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3566099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzjrcywp
2022-11-23T02:20:42.3566291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzjrcywp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3566542Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3566547Z 
2022-11-23T02:20:42.3566615Z Running tests...
2022-11-23T02:20:42.3566808Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3567027Z   test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16931
2022-11-23T02:20:42.3567179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16932
2022-11-23T02:20:42.3567322Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16933
2022-11-23T02:20:42.3567473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16934
2022-11-23T02:20:42.3567747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3567869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3568184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3568317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3568499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5ev2bgo
2022-11-23T02:20:42.3568694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5ev2bgo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3568952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3569143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3569421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3569552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3569735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1ljmge4
2022-11-23T02:20:42.3569927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1ljmge4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3570199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3570319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3570595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3570715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3570898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjz4tqx1
2022-11-23T02:20:42.3571089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjz4tqx1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3571358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3571479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3571758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3571888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3572069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ofywfyu
2022-11-23T02:20:42.3572250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ofywfyu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3572413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3572574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3572731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3572893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3573165Z [W tensorpipe_agent.cpp:939] RPC agent for worker1 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3573550Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down
2022-11-23T02:20:42.3573831Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down
2022-11-23T02:20:42.3574108Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down
2022-11-23T02:20:42.3574381Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down
2022-11-23T02:20:42.3574436Z ok (3.474s)
2022-11-23T02:20:42.3574441Z 
2022-11-23T02:20:42.3574666Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3574738Z Ran 1 test in 3.474s
2022-11-23T02:20:42.3574743Z 
2022-11-23T02:20:42.3574800Z OK
2022-11-23T02:20:42.3574805Z 
2022-11-23T02:20:42.3574970Z Generating XML reports...
2022-11-23T02:20:42.3575310Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021413.xml
2022-11-23T02:20:42.3575582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3575754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3576022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3576155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3576338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mtllkls
2022-11-23T02:20:42.3576531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mtllkls/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3576782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3576787Z 
2022-11-23T02:20:42.3576856Z Running tests...
2022-11-23T02:20:42.3577048Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3577274Z   test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17154
2022-11-23T02:20:42.3577426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17155
2022-11-23T02:20:42.3577566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17156
2022-11-23T02:20:42.3577716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17157
2022-11-23T02:20:42.3578150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3578271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3578549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3578682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3578865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ttpo5rj
2022-11-23T02:20:42.3579058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ttpo5rj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3579316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3579435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3579714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3579845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3580028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfgbkdtq
2022-11-23T02:20:42.3580223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfgbkdtq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3580487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3580604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3580885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3581007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3581188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxuehmj8
2022-11-23T02:20:42.3581492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxuehmj8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3581766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3581885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3582158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3582290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3582502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvx8z0_e
2022-11-23T02:20:42.3582695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvx8z0_e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3582846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3583007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3583166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3583323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3583413Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3583533Z ValueError('Expected error')
2022-11-23T02:20:42.3583622Z Traceback (most recent call last):
2022-11-23T02:20:42.3583870Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3584006Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3584351Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3584438Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3584521Z ValueError: Expected error
2022-11-23T02:20:42.3584526Z 
2022-11-23T02:20:42.3584614Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3584731Z ValueError('Expected error')
2022-11-23T02:20:42.3584818Z Traceback (most recent call last):
2022-11-23T02:20:42.3585062Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3585193Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3585469Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3585558Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3585638Z ValueError: Expected error
2022-11-23T02:20:42.3585643Z 
2022-11-23T02:20:42.3585728Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3585843Z ValueError('Expected error')
2022-11-23T02:20:42.3585926Z Traceback (most recent call last):
2022-11-23T02:20:42.3586171Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3586301Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3586571Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3586657Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3586735Z ValueError: Expected error
2022-11-23T02:20:42.3586740Z 
2022-11-23T02:20:42.3586824Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3586938Z ValueError('Expected error')
2022-11-23T02:20:42.3587025Z Traceback (most recent call last):
2022-11-23T02:20:42.3587268Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3587395Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3587668Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3587788Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3587869Z ValueError: Expected error
2022-11-23T02:20:42.3587874Z 
2022-11-23T02:20:42.3587960Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3588076Z ValueError('Expected error')
2022-11-23T02:20:42.3588152Z Traceback (most recent call last):
2022-11-23T02:20:42.3588399Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3588524Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3588794Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3588924Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3589003Z ValueError: Expected error
2022-11-23T02:20:42.3589007Z 
2022-11-23T02:20:42.3589090Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3589205Z ValueError('Expected error')
2022-11-23T02:20:42.3589283Z Traceback (most recent call last):
2022-11-23T02:20:42.3589537Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3589666Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3589943Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3590031Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3590111Z ValueError: Expected error
2022-11-23T02:20:42.3590115Z 
2022-11-23T02:20:42.3590200Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3590316Z ValueError('Expected error')
2022-11-23T02:20:42.3590393Z Traceback (most recent call last):
2022-11-23T02:20:42.3590644Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3590772Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3591156Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3591245Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3591323Z ValueError: Expected error
2022-11-23T02:20:42.3591328Z 
2022-11-23T02:20:42.3591412Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3591527Z ValueError('Expected error')
2022-11-23T02:20:42.3591605Z Traceback (most recent call last):
2022-11-23T02:20:42.3591857Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3591987Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3592259Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3592346Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3592426Z ValueError: Expected error
2022-11-23T02:20:42.3592431Z 
2022-11-23T02:20:42.3592519Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3592634Z ValueError('Expected error')
2022-11-23T02:20:42.3592710Z Traceback (most recent call last):
2022-11-23T02:20:42.3592957Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3593083Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3593359Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3593446Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3593528Z ValueError: Expected error
2022-11-23T02:20:42.3593532Z 
2022-11-23T02:20:42.3593617Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3593722Z ValueError('Expected error')
2022-11-23T02:20:42.3593808Z Traceback (most recent call last):
2022-11-23T02:20:42.3594053Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3594211Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3594480Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3594566Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3594648Z ValueError: Expected error
2022-11-23T02:20:42.3594653Z 
2022-11-23T02:20:42.3594738Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3594842Z ValueError('Expected error')
2022-11-23T02:20:42.3594930Z Traceback (most recent call last):
2022-11-23T02:20:42.3595174Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3595334Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3595606Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3595694Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3595777Z ValueError: Expected error
2022-11-23T02:20:42.3595782Z 
2022-11-23T02:20:42.3595868Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3595971Z ValueError('Expected error')
2022-11-23T02:20:42.3596059Z Traceback (most recent call last):
2022-11-23T02:20:42.3596517Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3596643Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3596916Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3597006Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3597087Z ValueError: Expected error
2022-11-23T02:20:42.3597091Z 
2022-11-23T02:20:42.3597176Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3597281Z ValueError('Expected error')
2022-11-23T02:20:42.3597366Z Traceback (most recent call last):
2022-11-23T02:20:42.3597620Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3597749Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3598018Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3598104Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3598185Z ValueError: Expected error
2022-11-23T02:20:42.3598189Z 
2022-11-23T02:20:42.3598275Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3598381Z ValueError('Expected error')
2022-11-23T02:20:42.3598467Z Traceback (most recent call last):
2022-11-23T02:20:42.3598718Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3598844Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3599116Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3599202Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3599282Z ValueError: Expected error
2022-11-23T02:20:42.3599287Z 
2022-11-23T02:20:42.3599361Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3599476Z ValueError('Expected error')
2022-11-23T02:20:42.3599562Z Traceback (most recent call last):
2022-11-23T02:20:42.3599809Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3599937Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3600207Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3600296Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3600375Z ValueError: Expected error
2022-11-23T02:20:42.3600380Z 
2022-11-23T02:20:42.3600454Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3600603Z ValueError('Expected error')
2022-11-23T02:20:42.3600693Z Traceback (most recent call last):
2022-11-23T02:20:42.3600940Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3601066Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3601407Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3601495Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3601574Z ValueError: Expected error
2022-11-23T02:20:42.3601608Z 
2022-11-23T02:20:42.3601684Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3601797Z ValueError('Expected error')
2022-11-23T02:20:42.3601884Z Traceback (most recent call last):
2022-11-23T02:20:42.3602130Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3602260Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3602532Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3602621Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3602703Z ValueError: Expected error
2022-11-23T02:20:42.3602707Z 
2022-11-23T02:20:42.3602783Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3602898Z ValueError('Expected error')
2022-11-23T02:20:42.3602984Z Traceback (most recent call last):
2022-11-23T02:20:42.3603231Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3603361Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3603632Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3603718Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3603789Z ValueError: Expected error
2022-11-23T02:20:42.3603806Z 
2022-11-23T02:20:42.3603880Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3603992Z ValueError('Expected error')
2022-11-23T02:20:42.3604077Z Traceback (most recent call last):
2022-11-23T02:20:42.3604324Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3604450Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3604721Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3604811Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3604881Z ValueError: Expected error
2022-11-23T02:20:42.3604963Z 
2022-11-23T02:20:42.3605045Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3605159Z ValueError('Expected error')
2022-11-23T02:20:42.3605244Z Traceback (most recent call last):
2022-11-23T02:20:42.3605494Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3605621Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3605893Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3605981Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3606051Z ValueError: Expected error
2022-11-23T02:20:42.3606056Z 
2022-11-23T02:20:42.3606141Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3606254Z ValueError('Expected error')
2022-11-23T02:20:42.3606342Z Traceback (most recent call last):
2022-11-23T02:20:42.3606657Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3606783Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3607087Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3607178Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3607249Z ValueError: Expected error
2022-11-23T02:20:42.3607254Z 
2022-11-23T02:20:42.3607340Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3607454Z ValueError('Expected error')
2022-11-23T02:20:42.3607541Z Traceback (most recent call last):
2022-11-23T02:20:42.3607787Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3607913Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3608211Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3608298Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3608368Z ValueError: Expected error
2022-11-23T02:20:42.3608373Z 
2022-11-23T02:20:42.3608458Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3608573Z ValueError('Expected error')
2022-11-23T02:20:42.3608665Z Traceback (most recent call last):
2022-11-23T02:20:42.3608908Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3609035Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3609303Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3609379Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3609460Z ValueError: Expected error
2022-11-23T02:20:42.3609464Z 
2022-11-23T02:20:42.3609551Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3609665Z ValueError('Expected error')
2022-11-23T02:20:42.3609753Z Traceback (most recent call last):
2022-11-23T02:20:42.3609997Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3610124Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3610395Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3610472Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3610553Z ValueError: Expected error
2022-11-23T02:20:42.3610558Z 
2022-11-23T02:20:42.3610644Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3610760Z ValueError('Expected error')
2022-11-23T02:20:42.3610853Z Traceback (most recent call last):
2022-11-23T02:20:42.3611100Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3611228Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3611497Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3611573Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3611655Z ValueError: Expected error
2022-11-23T02:20:42.3611660Z 
2022-11-23T02:20:42.3611747Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3611862Z ValueError('Expected error')
2022-11-23T02:20:42.3611949Z Traceback (most recent call last):
2022-11-23T02:20:42.3612192Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3612316Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3612582Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3612660Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3612739Z ValueError: Expected error
2022-11-23T02:20:42.3612744Z 
2022-11-23T02:20:42.3612830Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3612944Z ValueError('Expected error')
2022-11-23T02:20:42.3613033Z Traceback (most recent call last):
2022-11-23T02:20:42.3613307Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3613432Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3613690Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3613777Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3613858Z ValueError: Expected error
2022-11-23T02:20:42.3613862Z 
2022-11-23T02:20:42.3613949Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3614065Z ValueError('Expected error')
2022-11-23T02:20:42.3614152Z Traceback (most recent call last):
2022-11-23T02:20:42.3614427Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3614550Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3615063Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3615171Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3615251Z ValueError: Expected error
2022-11-23T02:20:42.3615256Z 
2022-11-23T02:20:42.3615337Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3615458Z ValueError('Expected error')
2022-11-23T02:20:42.3615545Z Traceback (most recent call last):
2022-11-23T02:20:42.3615795Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3615920Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3616178Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3616268Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3616349Z ValueError: Expected error
2022-11-23T02:20:42.3616354Z 
2022-11-23T02:20:42.3616437Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3616550Z ValueError('Expected error')
2022-11-23T02:20:42.3616639Z Traceback (most recent call last):
2022-11-23T02:20:42.3616887Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3617012Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3617269Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3617358Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3617439Z ValueError: Expected error
2022-11-23T02:20:42.3617443Z 
2022-11-23T02:20:42.3617528Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3617642Z ValueError('Expected error')
2022-11-23T02:20:42.3617730Z Traceback (most recent call last):
2022-11-23T02:20:42.3617977Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3618092Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3618360Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3618449Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3618531Z ValueError: Expected error
2022-11-23T02:20:42.3618535Z 
2022-11-23T02:20:42.3618619Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3618732Z ValueError('Expected error')
2022-11-23T02:20:42.3618818Z Traceback (most recent call last):
2022-11-23T02:20:42.3619064Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3619180Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3619447Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3619533Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3619614Z ValueError: Expected error
2022-11-23T02:20:42.3619618Z 
2022-11-23T02:20:42.3619758Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3619874Z ValueError('Expected error')
2022-11-23T02:20:42.3619961Z Traceback (most recent call last):
2022-11-23T02:20:42.3620208Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3620323Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3620591Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3620679Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3620798Z ValueError: Expected error
2022-11-23T02:20:42.3620802Z 
2022-11-23T02:20:42.3620885Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3621001Z ValueError('Expected error')
2022-11-23T02:20:42.3621086Z Traceback (most recent call last):
2022-11-23T02:20:42.3621329Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3621447Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3621714Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3621799Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3621881Z ValueError: Expected error
2022-11-23T02:20:42.3621886Z 
2022-11-23T02:20:42.3621973Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3622204Z ValueError('Expected error')
2022-11-23T02:20:42.3622291Z Traceback (most recent call last):
2022-11-23T02:20:42.3622533Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3622660Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3622928Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3623019Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3623101Z ValueError: Expected error
2022-11-23T02:20:42.3623105Z 
2022-11-23T02:20:42.3623190Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3623305Z ValueError('Expected error')
2022-11-23T02:20:42.3623391Z Traceback (most recent call last):
2022-11-23T02:20:42.3623627Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3623752Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3624018Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3624107Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3624188Z ValueError: Expected error
2022-11-23T02:20:42.3624193Z 
2022-11-23T02:20:42.3624276Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3624389Z ValueError('Expected error')
2022-11-23T02:20:42.3624477Z Traceback (most recent call last):
2022-11-23T02:20:42.3624718Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3624843Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3625112Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3625200Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3625281Z ValueError: Expected error
2022-11-23T02:20:42.3625285Z 
2022-11-23T02:20:42.3625369Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3625555Z ValueError('Expected error')
2022-11-23T02:20:42.3625642Z Traceback (most recent call last):
2022-11-23T02:20:42.3625880Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3626005Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3626305Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3626395Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3626477Z ValueError: Expected error
2022-11-23T02:20:42.3626481Z 
2022-11-23T02:20:42.3626567Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3626682Z ValueError('Expected error')
2022-11-23T02:20:42.3626758Z Traceback (most recent call last):
2022-11-23T02:20:42.3627006Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3627130Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3627426Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3627509Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3627591Z ValueError: Expected error
2022-11-23T02:20:42.3627595Z 
2022-11-23T02:20:42.3627681Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3627798Z ValueError('Expected error')
2022-11-23T02:20:42.3627873Z Traceback (most recent call last):
2022-11-23T02:20:42.3628120Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3628305Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3628576Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3628659Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3628738Z ValueError: Expected error
2022-11-23T02:20:42.3628745Z 
2022-11-23T02:20:42.3628809Z ok (2.474s)
2022-11-23T02:20:42.3628814Z 
2022-11-23T02:20:42.3629009Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3629071Z Ran 1 test in 2.474s
2022-11-23T02:20:42.3629085Z 
2022-11-23T02:20:42.3629132Z OK
2022-11-23T02:20:42.3629137Z 
2022-11-23T02:20:42.3629217Z Generating XML reports...
2022-11-23T02:20:42.3629560Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021418.xml
2022-11-23T02:20:42.3629831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3629953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3630231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3630365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3630563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0uq4qq0
2022-11-23T02:20:42.3630746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0uq4qq0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3630999Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3631006Z 
2022-11-23T02:20:42.3631075Z Running tests...
2022-11-23T02:20:42.3631268Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3631499Z   test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17377
2022-11-23T02:20:42.3631650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17378
2022-11-23T02:20:42.3631801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17379
2022-11-23T02:20:42.3631956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17380
2022-11-23T02:20:42.3632227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3632337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3632646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3632780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3632962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvd7r0mr
2022-11-23T02:20:42.3633151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvd7r0mr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3633422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3633542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3633867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3633987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3634168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9w_9p3tn
2022-11-23T02:20:42.3634355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9w_9p3tn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3634623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3634742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3635015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3635143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3635325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqxm3bjd
2022-11-23T02:20:42.3635512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqxm3bjd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3635767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3635888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3636162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3636291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3636471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzw3bf99
2022-11-23T02:20:42.3636659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzw3bf99/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3636824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3636986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3637135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3637296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3637438Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:20:42.3637555Z ValueError('Expected error')
2022-11-23T02:20:42.3637642Z Traceback (most recent call last):
2022-11-23T02:20:42.3637900Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3638033Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3638315Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3638396Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3638477Z ValueError: Expected error
2022-11-23T02:20:42.3638483Z 
2022-11-23T02:20:42.3638568Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:20:42.3638684Z ValueError('Expected error')
2022-11-23T02:20:42.3638772Z Traceback (most recent call last):
2022-11-23T02:20:42.3639059Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3639188Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3639583Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3639661Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3639743Z ValueError: Expected error
2022-11-23T02:20:42.3639748Z 
2022-11-23T02:20:42.3639833Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:20:42.3639950Z ValueError('Expected error')
2022-11-23T02:20:42.3640071Z Traceback (most recent call last):
2022-11-23T02:20:42.3640321Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3640448Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3640721Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3640800Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3640881Z ValueError: Expected error
2022-11-23T02:20:42.3640885Z 
2022-11-23T02:20:42.3640970Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:20:42.3641085Z ValueError('Expected error')
2022-11-23T02:20:42.3641169Z Traceback (most recent call last):
2022-11-23T02:20:42.3641418Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:20:42.3641541Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:20:42.3641805Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 406, in raise_func
2022-11-23T02:20:42.3641890Z     raise ValueError(expected_err)
2022-11-23T02:20:42.3641970Z ValueError: Expected error
2022-11-23T02:20:42.3641975Z 
2022-11-23T02:20:42.3642039Z ok (2.476s)
2022-11-23T02:20:42.3642045Z 
2022-11-23T02:20:42.3642240Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3642311Z Ran 1 test in 2.476s
2022-11-23T02:20:42.3642316Z 
2022-11-23T02:20:42.3642371Z OK
2022-11-23T02:20:42.3642376Z 
2022-11-23T02:20:42.3642456Z Generating XML reports...
2022-11-23T02:20:42.3642790Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021422.xml
2022-11-23T02:20:42.3643051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3643171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3643521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3643652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3643833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l98jb3g
2022-11-23T02:20:42.3644026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l98jb3g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3644278Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3644283Z 
2022-11-23T02:20:42.3644350Z Running tests...
2022-11-23T02:20:42.3644531Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3644754Z   test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17600
2022-11-23T02:20:42.3644970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17601
2022-11-23T02:20:42.3645126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17602
2022-11-23T02:20:42.3645274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17603
2022-11-23T02:20:42.3645585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3645709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3645989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3646121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3646294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwstq86f
2022-11-23T02:20:42.3646488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwstq86f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3646789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3646911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3647191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3647323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3647503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79bv1o4d
2022-11-23T02:20:42.3647692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79bv1o4d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3647959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3648070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3648349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3648482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3648668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyppvbcn
2022-11-23T02:20:42.3648866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyppvbcn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3649132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3649318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3649594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3649715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3649894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig4z29f2
2022-11-23T02:20:42.3650087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig4z29f2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3650249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3650410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3650571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3650729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3650904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3651080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3651240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3651412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3651710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3652038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3652330Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3652617Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3652881Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3653142Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3653426Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3653685Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3653936Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3654178Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3654244Z ok (2.777s)
2022-11-23T02:20:42.3654250Z 
2022-11-23T02:20:42.3654445Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3654518Z Ran 1 test in 2.777s
2022-11-23T02:20:42.3654523Z 
2022-11-23T02:20:42.3654750Z OK
2022-11-23T02:20:42.3654755Z 
2022-11-23T02:20:42.3654835Z Generating XML reports...
2022-11-23T02:20:42.3655264Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021426.xml
2022-11-23T02:20:42.3655540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3655664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3655933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3656063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3656252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzhs57ku
2022-11-23T02:20:42.3656443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzhs57ku/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3656699Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3656705Z 
2022-11-23T02:20:42.3656773Z Running tests...
2022-11-23T02:20:42.3656969Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3657191Z   test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17835
2022-11-23T02:20:42.3657334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17836
2022-11-23T02:20:42.3657484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17837
2022-11-23T02:20:42.3657636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17838
2022-11-23T02:20:42.3657911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3658030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3658309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3658495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3658680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zwwbmmc
2022-11-23T02:20:42.3658875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zwwbmmc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3659132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3659251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3659571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3659708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3659891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1uxko26
2022-11-23T02:20:42.3660082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1uxko26/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3660349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3660467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3660735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3660865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3661045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc1_8jqk
2022-11-23T02:20:42.3661237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc1_8jqk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3661503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3661621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3661898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3662029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3662211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwlzuq54
2022-11-23T02:20:42.3662389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwlzuq54/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3662549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3662709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3662866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3663025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3663200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3663374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3663544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3663714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3663999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3664294Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3664581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3664912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3665175Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3665442Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3665789Z [W tensorpipe_agent.cpp:939] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3666077Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3666329Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3666588Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3666848Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3667109Z [W tensorpipe_agent.cpp:915] RPC agent for worker0 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3667381Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #3 to worker0, as the agent is shutting down
2022-11-23T02:20:42.3667645Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3667900Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3668150Z [W tensorpipe_agent.cpp:678] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3668217Z ok (5.578s)
2022-11-23T02:20:42.3668223Z 
2022-11-23T02:20:42.3668419Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3668492Z Ran 1 test in 5.578s
2022-11-23T02:20:42.3668498Z 
2022-11-23T02:20:42.3668553Z OK
2022-11-23T02:20:42.3668558Z 
2022-11-23T02:20:42.3668637Z Generating XML reports...
2022-11-23T02:20:42.3668975Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021431.xml
2022-11-23T02:20:42.3669239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3669359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3669818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3669952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3670141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi2ulg0i
2022-11-23T02:20:42.3670337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi2ulg0i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3670593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3670599Z 
2022-11-23T02:20:42.3670740Z Running tests...
2022-11-23T02:20:42.3670924Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3671155Z   test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18070
2022-11-23T02:20:42.3671308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18071
2022-11-23T02:20:42.3671460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18072
2022-11-23T02:20:42.3671606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18073
2022-11-23T02:20:42.3671907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3672027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3672310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3672445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3672616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglexh0ll
2022-11-23T02:20:42.3672813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglexh0ll/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3673084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3673203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3673560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3673692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3673871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyq5ebny
2022-11-23T02:20:42.3674064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyq5ebny/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3674333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3674442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3674719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3674849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3675026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_bdng1e
2022-11-23T02:20:42.3675219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_bdng1e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3675485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3675607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3675883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3676003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3676245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1kn088e
2022-11-23T02:20:42.3676435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1kn088e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3676598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3676759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3676919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3677075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3677280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3677457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3677618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3677786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3678087Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3678425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3678714Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3679005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3679266Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3679524Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3679793Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.3680045Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3680298Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3680542Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.3680609Z ok (2.674s)
2022-11-23T02:20:42.3680614Z 
2022-11-23T02:20:42.3680809Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3680882Z Ran 1 test in 2.674s
2022-11-23T02:20:42.3680887Z 
2022-11-23T02:20:42.3680945Z OK
2022-11-23T02:20:42.3680949Z 
2022-11-23T02:20:42.3681030Z Generating XML reports...
2022-11-23T02:20:42.3681367Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021439.xml
2022-11-23T02:20:42.3681641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3681763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3682030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3682163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3682347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivqpje2x
2022-11-23T02:20:42.3682543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivqpje2x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3682797Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3682802Z 
2022-11-23T02:20:42.3682872Z Running tests...
2022-11-23T02:20:42.3683065Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3683297Z   test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18305
2022-11-23T02:20:42.3683440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18306
2022-11-23T02:20:42.3683592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18307
2022-11-23T02:20:42.3683739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18308
2022-11-23T02:20:42.3684011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3684128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3684435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3684569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3684751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3k_ubz3
2022-11-23T02:20:42.3685023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3k_ubz3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3685284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3685403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3685679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3685811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3686117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjam_oh7
2022-11-23T02:20:42.3686312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjam_oh7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3686584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3686705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3686970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3687103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3687284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiclvibl
2022-11-23T02:20:42.3687475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiclvibl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3687740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3687861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3688139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3688272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3688459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpti0ovg
2022-11-23T02:20:42.3688641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpti0ovg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3688801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3688961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3689121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3689279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3689342Z ok (2.478s)
2022-11-23T02:20:42.3689348Z 
2022-11-23T02:20:42.3689540Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3689612Z Ran 1 test in 2.478s
2022-11-23T02:20:42.3689617Z 
2022-11-23T02:20:42.3689664Z OK
2022-11-23T02:20:42.3689678Z 
2022-11-23T02:20:42.3689784Z Generating XML reports...
2022-11-23T02:20:42.3690121Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021443.xml
2022-11-23T02:20:42.3690395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3690515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3690792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3690956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3691137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9qyddug
2022-11-23T02:20:42.3691332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9qyddug/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3691576Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3691581Z 
2022-11-23T02:20:42.3691651Z Running tests...
2022-11-23T02:20:42.3691845Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3692059Z   test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18528
2022-11-23T02:20:42.3692212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18529
2022-11-23T02:20:42.3692363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18530
2022-11-23T02:20:42.3692516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18531
2022-11-23T02:20:42.3692788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3692897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3693178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3693309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3693491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnbqkkx6
2022-11-23T02:20:42.3693686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnbqkkx6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3693955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3694074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3694351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3694481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3694652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfru6t_j6
2022-11-23T02:20:42.3694845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfru6t_j6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3695265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3695385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3695660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3695791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3695972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_b1jce6r
2022-11-23T02:20:42.3696163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_b1jce6r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3696494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3696606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3696877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3697006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3697186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3s9vuag
2022-11-23T02:20:42.3697377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3s9vuag/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3697578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3697736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3697894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3698043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3698106Z ok (2.580s)
2022-11-23T02:20:42.3698111Z 
2022-11-23T02:20:42.3698305Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3698506Z Ran 1 test in 2.581s
2022-11-23T02:20:42.3698511Z 
2022-11-23T02:20:42.3698569Z OK
2022-11-23T02:20:42.3698574Z 
2022-11-23T02:20:42.3698652Z Generating XML reports...
2022-11-23T02:20:42.3698994Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021447.xml
2022-11-23T02:20:42.3699266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3699387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3699651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3699786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3699970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqh9mauu
2022-11-23T02:20:42.3700165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqh9mauu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3700416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3700422Z 
2022-11-23T02:20:42.3700491Z Running tests...
2022-11-23T02:20:42.3700686Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3700901Z   test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18751
2022-11-23T02:20:42.3706253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18752
2022-11-23T02:20:42.3706453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18753
2022-11-23T02:20:42.3706617Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18754
2022-11-23T02:20:42.3706905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3707028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3707311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3707444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3707631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2apqizv
2022-11-23T02:20:42.3707828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2apqizv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3708103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3708294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3708576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3708696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3708880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput8jopdq
2022-11-23T02:20:42.3709073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput8jopdq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3709343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3709499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3709780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3709914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3710097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbt4pxtw
2022-11-23T02:20:42.3710280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbt4pxtw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3710553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3710672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3710950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3711083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3711265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqx4l81n
2022-11-23T02:20:42.3711455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqx4l81n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3711619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3711782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3711929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3712091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3712155Z ok (2.373s)
2022-11-23T02:20:42.3712162Z 
2022-11-23T02:20:42.3712361Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3712436Z Ran 1 test in 2.373s
2022-11-23T02:20:42.3712442Z 
2022-11-23T02:20:42.3712501Z OK
2022-11-23T02:20:42.3712505Z 
2022-11-23T02:20:42.3712588Z Generating XML reports...
2022-11-23T02:20:42.3712930Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021452.xml
2022-11-23T02:20:42.3713331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3713450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3713732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3713865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3714048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrma4q0e
2022-11-23T02:20:42.3714240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrma4q0e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3714498Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3714504Z 
2022-11-23T02:20:42.3714575Z Running tests...
2022-11-23T02:20:42.3714770Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3715005Z   test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18848
2022-11-23T02:20:42.3715162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18849
2022-11-23T02:20:42.3715312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18850
2022-11-23T02:20:42.3715462Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18851
2022-11-23T02:20:42.3715730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3715879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3716160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3716290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3716463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv88lufb
2022-11-23T02:20:42.3716655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv88lufb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3716925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3717045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3717321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3717452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3717637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg05ue8lv
2022-11-23T02:20:42.3717828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg05ue8lv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3718101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3718210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3718488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3718617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3718796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68t25h8t
2022-11-23T02:20:42.3718985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68t25h8t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3719254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3719376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3719653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3719786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3720061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplem9c83t
2022-11-23T02:20:42.3720254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplem9c83t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3720413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3720575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3720736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3720895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3720966Z ok (2.274s)
2022-11-23T02:20:42.3720971Z 
2022-11-23T02:20:42.3721169Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3721263Z Ran 1 test in 2.275s
2022-11-23T02:20:42.3721268Z 
2022-11-23T02:20:42.3721327Z OK
2022-11-23T02:20:42.3721332Z 
2022-11-23T02:20:42.3721416Z Generating XML reports...
2022-11-23T02:20:42.3721755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021456.xml
2022-11-23T02:20:42.3722025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3722146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3722426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3722587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3722766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9e_vh1k
2022-11-23T02:20:42.3722946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9e_vh1k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3723196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3723201Z 
2022-11-23T02:20:42.3723271Z Running tests...
2022-11-23T02:20:42.3723463Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3723755Z   test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18903
2022-11-23T02:20:42.3723911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18904
2022-11-23T02:20:42.3724067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18905
2022-11-23T02:20:42.3724218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18906
2022-11-23T02:20:42.3724480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3724604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3724884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3725106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3725290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1dsaf2mo
2022-11-23T02:20:42.3725560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1dsaf2mo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3725842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3725966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3726247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3726370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3726555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx80izr5
2022-11-23T02:20:42.3726748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx80izr5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3727018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3727140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3727418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3727555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3727740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz48ks3zo
2022-11-23T02:20:42.3727956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz48ks3zo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3728227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3728346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3728623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3728754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3728935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41ohoq2w
2022-11-23T02:20:42.3729171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41ohoq2w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3729333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3729494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3729642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3729802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3729931Z ok (2.476s)
2022-11-23T02:20:42.3729937Z 
2022-11-23T02:20:42.3730136Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3730209Z Ran 1 test in 2.476s
2022-11-23T02:20:42.3730214Z 
2022-11-23T02:20:42.3730271Z OK
2022-11-23T02:20:42.3730275Z 
2022-11-23T02:20:42.3730357Z Generating XML reports...
2022-11-23T02:20:42.3730789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021500.xml
2022-11-23T02:20:42.3731058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3731168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3731450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3731584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3731765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs67tv3o
2022-11-23T02:20:42.3731954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs67tv3o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3732206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3732213Z 
2022-11-23T02:20:42.3732283Z Running tests...
2022-11-23T02:20:42.3732477Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3732760Z   test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19158
2022-11-23T02:20:42.3732917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19159
2022-11-23T02:20:42.3733068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19160
2022-11-23T02:20:42.3733215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19161
2022-11-23T02:20:42.3733486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3733605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3733886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3734020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3734203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphngejznf
2022-11-23T02:20:42.3734418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphngejznf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3734682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3734803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3735253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3735383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3735564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mpevy7b
2022-11-23T02:20:42.3735809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mpevy7b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3736079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3736187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3736461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3736592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3736770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6txnicd5
2022-11-23T02:20:42.3736963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6txnicd5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3737231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3737359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3737633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3737764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3737934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_t0x8bs9
2022-11-23T02:20:42.3738123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_t0x8bs9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3738282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3738526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3738687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3738846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3738914Z ok (2.775s)
2022-11-23T02:20:42.3738919Z 
2022-11-23T02:20:42.3739114Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3739174Z Ran 1 test in 2.776s
2022-11-23T02:20:42.3739188Z 
2022-11-23T02:20:42.3739235Z OK
2022-11-23T02:20:42.3739243Z 
2022-11-23T02:20:42.3739324Z Generating XML reports...
2022-11-23T02:20:42.3739754Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021505.xml
2022-11-23T02:20:42.3740028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3740151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3740429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3740564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3740746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ceunqrb
2022-11-23T02:20:42.3740928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ceunqrb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3741220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3741226Z 
2022-11-23T02:20:42.3741297Z Running tests...
2022-11-23T02:20:42.3741492Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3741782Z   test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19458
2022-11-23T02:20:42.3741935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19459
2022-11-23T02:20:42.3742086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19460
2022-11-23T02:20:42.3742269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19461
2022-11-23T02:20:42.3742540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3742650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3742932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3743062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3743244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpbnaac6
2022-11-23T02:20:42.3743437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpbnaac6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3743713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3743839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3744119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3744240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3744426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbns_ani
2022-11-23T02:20:42.3744620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbns_ani/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3744891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3745012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3745290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3745422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3745605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnupdx95h
2022-11-23T02:20:42.3745798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnupdx95h/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3746057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3746176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3746450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3746584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3746766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvqv7rt4
2022-11-23T02:20:42.3746958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvqv7rt4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3747124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3747284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3747433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3747621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3747687Z ok (2.578s)
2022-11-23T02:20:42.3747692Z 
2022-11-23T02:20:42.3747889Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3747964Z Ran 1 test in 2.579s
2022-11-23T02:20:42.3747969Z 
2022-11-23T02:20:42.3748027Z OK
2022-11-23T02:20:42.3748031Z 
2022-11-23T02:20:42.3748114Z Generating XML reports...
2022-11-23T02:20:42.3748536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021509.xml
2022-11-23T02:20:42.3748835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3748945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3749225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3749356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3749613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvssmu70
2022-11-23T02:20:42.3749810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvssmu70/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3750066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3750071Z 
2022-11-23T02:20:42.3750142Z Running tests...
2022-11-23T02:20:42.3750336Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3750623Z   test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19713
2022-11-23T02:20:42.3750767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19714
2022-11-23T02:20:42.3750923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19715
2022-11-23T02:20:42.3751072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19716
2022-11-23T02:20:42.3751344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3751463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3751744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3751875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3752059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsipmiqmi
2022-11-23T02:20:42.3752242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsipmiqmi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3752511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3752630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3752902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3753033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3753215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmubgtmc
2022-11-23T02:20:42.3753408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmubgtmc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3753678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3753797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3754060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3754220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3754401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueillsv5
2022-11-23T02:20:42.3754595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueillsv5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3754864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3754985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3755330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3755494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3755662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_87o55d
2022-11-23T02:20:42.3755853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_87o55d/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3756014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3756173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3756332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3756490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3756555Z ok (2.473s)
2022-11-23T02:20:42.3756560Z 
2022-11-23T02:20:42.3756759Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3756832Z Ran 1 test in 2.473s
2022-11-23T02:20:42.3756837Z 
2022-11-23T02:20:42.3756883Z OK
2022-11-23T02:20:42.3756888Z 
2022-11-23T02:20:42.3756970Z Generating XML reports...
2022-11-23T02:20:42.3757395Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021514.xml
2022-11-23T02:20:42.3757668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3757792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3758072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3758202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3758387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6a4_ka8v
2022-11-23T02:20:42.3758568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6a4_ka8v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3758820Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3758828Z 
2022-11-23T02:20:42.3758899Z Running tests...
2022-11-23T02:20:42.3759094Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3759381Z   test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19983
2022-11-23T02:20:42.3759536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19984
2022-11-23T02:20:42.3759689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19985
2022-11-23T02:20:42.3759838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19986
2022-11-23T02:20:42.3760107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3760216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3760497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3760656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3760841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmvutb8o
2022-11-23T02:20:42.3761034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmvutb8o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3761307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3761429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3761709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3761928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3762100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z12be09
2022-11-23T02:20:42.3762293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z12be09/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3762559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3762678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3762956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3763153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3763335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqryrwlp
2022-11-23T02:20:42.3763530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqryrwlp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3763788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3763907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3764184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3764315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3764495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20d6yssq
2022-11-23T02:20:42.3764687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20d6yssq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3764882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3765112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3765275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3765422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3765487Z ok (2.574s)
2022-11-23T02:20:42.3765495Z 
2022-11-23T02:20:42.3765693Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3765763Z Ran 1 test in 2.575s
2022-11-23T02:20:42.3765767Z 
2022-11-23T02:20:42.3765830Z OK
2022-11-23T02:20:42.3765834Z 
2022-11-23T02:20:42.3765915Z Generating XML reports...
2022-11-23T02:20:42.3766338Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021518.xml
2022-11-23T02:20:42.3766611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3766723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3767000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3767133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3767415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80zej3e9
2022-11-23T02:20:42.3767610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80zej3e9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3767864Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3767869Z 
2022-11-23T02:20:42.3767938Z Running tests...
2022-11-23T02:20:42.3768130Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3768407Z   test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20262
2022-11-23T02:20:42.3768581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20263
2022-11-23T02:20:42.3768732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20264
2022-11-23T02:20:42.3768889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20265
2022-11-23T02:20:42.3769164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3769286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3769568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3769701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3769884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit8778h2
2022-11-23T02:20:42.3770081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit8778h2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3770340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3770462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3770742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3770881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3771065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih6s2qk7
2022-11-23T02:20:42.3771258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih6s2qk7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3771528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3771646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3771912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3772044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3772226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyb78pd5
2022-11-23T02:20:42.3772421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyb78pd5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3772690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3772810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3773084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3773214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3773397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_gl6lik
2022-11-23T02:20:42.3773576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_gl6lik/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3773762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3773926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3774085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3774242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3774306Z ok (2.576s)
2022-11-23T02:20:42.3774312Z 
2022-11-23T02:20:42.3774508Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3774578Z Ran 1 test in 2.576s
2022-11-23T02:20:42.3774658Z 
2022-11-23T02:20:42.3774705Z OK
2022-11-23T02:20:42.3774720Z 
2022-11-23T02:20:42.3774790Z Generating XML reports...
2022-11-23T02:20:42.3775383Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021522.xml
2022-11-23T02:20:42.3775660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3775783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3776062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3776281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3776464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptcqpefv
2022-11-23T02:20:42.3776659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptcqpefv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3776905Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3776922Z 
2022-11-23T02:20:42.3776981Z Running tests...
2022-11-23T02:20:42.3777177Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3777468Z   test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20541
2022-11-23T02:20:42.3777623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20542
2022-11-23T02:20:42.3777775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20543
2022-11-23T02:20:42.3777924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20544
2022-11-23T02:20:42.3778195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3778316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3778584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3778719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3778907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprsheciz1
2022-11-23T02:20:42.3779102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprsheciz1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3779376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3779499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3779779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3779911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3780084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjsb41g44
2022-11-23T02:20:42.3780355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3780476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3780808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3780941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3781122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mc2u6np
2022-11-23T02:20:42.3781315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjsb41g44/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3781503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mc2u6np/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3781808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3781918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3782192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3782323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3782506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hka3ivh
2022-11-23T02:20:42.3782695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hka3ivh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3782857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3783021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3783181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3783329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3783392Z ok (2.573s)
2022-11-23T02:20:42.3783397Z 
2022-11-23T02:20:42.3783588Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3783662Z Ran 1 test in 2.573s
2022-11-23T02:20:42.3783667Z 
2022-11-23T02:20:42.3783723Z OK
2022-11-23T02:20:42.3783728Z 
2022-11-23T02:20:42.3783807Z Generating XML reports...
2022-11-23T02:20:42.3784232Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021527.xml
2022-11-23T02:20:42.3784500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3784621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3784887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3785021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3785203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g9ka1sy
2022-11-23T02:20:42.3785396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g9ka1sy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3785648Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3785654Z 
2022-11-23T02:20:42.3785722Z Running tests...
2022-11-23T02:20:42.3785913Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3786204Z   test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20796
2022-11-23T02:20:42.3786357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20797
2022-11-23T02:20:42.3786501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20798
2022-11-23T02:20:42.3786652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20799
2022-11-23T02:20:42.3786950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3787074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3787351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3787484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3787797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1tqx7g8
2022-11-23T02:20:42.3787988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1tqx7g8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3788282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3788403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3788681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3788814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3788997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyahl1mwa
2022-11-23T02:20:42.3789192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyahl1mwa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3789462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3789582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3789857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3789979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3790160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2immcmv
2022-11-23T02:20:42.3790355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2immcmv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3790623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3790741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3791016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3791146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3791397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9qz2rqx
2022-11-23T02:20:42.3791582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9qz2rqx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3791744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3791905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3792066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3792230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3792295Z ok (2.576s)
2022-11-23T02:20:42.3792300Z 
2022-11-23T02:20:42.3792494Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3792568Z Ran 1 test in 2.576s
2022-11-23T02:20:42.3792573Z 
2022-11-23T02:20:42.3792631Z OK
2022-11-23T02:20:42.3792636Z 
2022-11-23T02:20:42.3792705Z Generating XML reports...
2022-11-23T02:20:42.3793129Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021531.xml
2022-11-23T02:20:42.3793402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3793524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3793834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3793967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3794150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp85bw9u
2022-11-23T02:20:42.3794344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp85bw9u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3794585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3794630Z 
2022-11-23T02:20:42.3794690Z Running tests...
2022-11-23T02:20:42.3794886Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3795173Z   test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21051
2022-11-23T02:20:42.3795329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21052
2022-11-23T02:20:42.3795480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21053
2022-11-23T02:20:42.3795633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21054
2022-11-23T02:20:42.3795976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3796095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3796363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3796498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3796680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuslgkwl
2022-11-23T02:20:42.3796877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuslgkwl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3797148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3797267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3797540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3797671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3797851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmjav5sx
2022-11-23T02:20:42.3798033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmjav5sx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3798302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3798422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3798706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3798841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3799021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg67whipq
2022-11-23T02:20:42.3799212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg67whipq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3799480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3799589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3799867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3800002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3800219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4a0vj2q
2022-11-23T02:20:42.3800411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4a0vj2q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3800573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3800736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3800896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3801055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3801138Z ok (2.475s)
2022-11-23T02:20:42.3801143Z 
2022-11-23T02:20:42.3801337Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3801410Z Ran 1 test in 2.475s
2022-11-23T02:20:42.3801415Z 
2022-11-23T02:20:42.3801472Z OK
2022-11-23T02:20:42.3801477Z 
2022-11-23T02:20:42.3801559Z Generating XML reports...
2022-11-23T02:20:42.3801979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021536.xml
2022-11-23T02:20:42.3802248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3802368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3802636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3802771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3802950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuggjzuua
2022-11-23T02:20:42.3803144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuggjzuua/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3803396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3803402Z 
2022-11-23T02:20:42.3803472Z Running tests...
2022-11-23T02:20:42.3803663Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3803944Z   test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21306
2022-11-23T02:20:42.3804098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21307
2022-11-23T02:20:42.3804239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21308
2022-11-23T02:20:42.3804392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21309
2022-11-23T02:20:42.3804666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3804787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3805145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3805280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3805462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_bgc5wu
2022-11-23T02:20:42.3805653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_bgc5wu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3805927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3806037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3806317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3806449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3806665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqpxvwe5
2022-11-23T02:20:42.3806862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqpxvwe5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3807133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3807252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3807529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3807649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3807860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv32sqocp
2022-11-23T02:20:42.3808054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv32sqocp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3808323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3808444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3808723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3808856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3809036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfqydamg
2022-11-23T02:20:42.3809231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfqydamg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3809383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3809545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3809706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3809866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3809931Z ok (2.673s)
2022-11-23T02:20:42.3809936Z 
2022-11-23T02:20:42.3810256Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3810325Z Ran 1 test in 2.674s
2022-11-23T02:20:42.3810330Z 
2022-11-23T02:20:42.3810390Z OK
2022-11-23T02:20:42.3810395Z 
2022-11-23T02:20:42.3810465Z Generating XML reports...
2022-11-23T02:20:42.3810886Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021540.xml
2022-11-23T02:20:42.3811160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3811282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3811559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3811693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3811879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppentqwb_
2022-11-23T02:20:42.3812074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppentqwb_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3812327Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3812332Z 
2022-11-23T02:20:42.3812391Z Running tests...
2022-11-23T02:20:42.3812587Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3812881Z   test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21561
2022-11-23T02:20:42.3813034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21562
2022-11-23T02:20:42.3813219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21563
2022-11-23T02:20:42.3813371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21564
2022-11-23T02:20:42.3813641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3813761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3814041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3814163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3814377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmmlla6r
2022-11-23T02:20:42.3814571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmmlla6r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3814846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3815112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3815390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3815655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3815841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuou9hrv
2022-11-23T02:20:42.3816022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuou9hrv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3816301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3816420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3816701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3816835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3817015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7w2eu9k
2022-11-23T02:20:42.3817207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7w2eu9k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3817472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3817592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3817858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3817992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3818171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hv6t__m
2022-11-23T02:20:42.3818363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hv6t__m/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3818524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3818686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3818846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3819002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3819120Z ok (2.580s)
2022-11-23T02:20:42.3819126Z 
2022-11-23T02:20:42.3819329Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3819402Z Ran 1 test in 2.581s
2022-11-23T02:20:42.3819407Z 
2022-11-23T02:20:42.3819464Z OK
2022-11-23T02:20:42.3819468Z 
2022-11-23T02:20:42.3819549Z Generating XML reports...
2022-11-23T02:20:42.3820029Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021544.xml
2022-11-23T02:20:42.3820304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3820423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3820704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3820826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3821010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq61mhqdn
2022-11-23T02:20:42.3821259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq61mhqdn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3821514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3821519Z 
2022-11-23T02:20:42.3821587Z Running tests...
2022-11-23T02:20:42.3821782Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3822074Z   test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21816
2022-11-23T02:20:42.3822226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21817
2022-11-23T02:20:42.3822377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21818
2022-11-23T02:20:42.3822516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21819
2022-11-23T02:20:42.3822793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3822911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3823191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3823327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3823510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0suhz7e7
2022-11-23T02:20:42.3823704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0suhz7e7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3823977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3824086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3824363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3824498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3824678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx5g4229
2022-11-23T02:20:42.3824871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx5g4229/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3825141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3825261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3825536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3825669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3825840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8o81plq
2022-11-23T02:20:42.3826034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8o81plq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3826421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3826543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3826855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3826991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3827171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufp6z3r2
2022-11-23T02:20:42.3827365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufp6z3r2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3827513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3827705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3827867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3828028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3828205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3828379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3828549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3828720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3829020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3829308Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3829599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3830044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3830110Z ok (2.873s)
2022-11-23T02:20:42.3830115Z 
2022-11-23T02:20:42.3830314Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3830387Z Ran 1 test in 2.873s
2022-11-23T02:20:42.3830392Z 
2022-11-23T02:20:42.3830449Z OK
2022-11-23T02:20:42.3830453Z 
2022-11-23T02:20:42.3830535Z Generating XML reports...
2022-11-23T02:20:42.3830964Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021549.xml
2022-11-23T02:20:42.3831227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3831348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3831630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3831764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3831948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lfnddyw
2022-11-23T02:20:42.3832140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lfnddyw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3832393Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3832398Z 
2022-11-23T02:20:42.3832468Z Running tests...
2022-11-23T02:20:42.3832661Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3833011Z   test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22063
2022-11-23T02:20:42.3833168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22064
2022-11-23T02:20:42.3833354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22065
2022-11-23T02:20:42.3833505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22066
2022-11-23T02:20:42.3833780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3833900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3834181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3834314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3834514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_agpq7_e
2022-11-23T02:20:42.3834713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_agpq7_e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3834984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3835109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3835387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3835520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3835704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmple3u4hh9
2022-11-23T02:20:42.3835897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmple3u4hh9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3836169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3836278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3836556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3836688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3836867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8611ipyg
2022-11-23T02:20:42.3837057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8611ipyg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3837391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3837511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3837788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3837920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3838091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbowh5uh
2022-11-23T02:20:42.3838287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbowh5uh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3838449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3838612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3838773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3838931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3839107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3839283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3839573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3839776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3839946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3840240Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3840536Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3840820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3840913Z ok (2.678s)
2022-11-23T02:20:42.3840919Z 
2022-11-23T02:20:42.3841117Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3841189Z Ran 1 test in 2.678s
2022-11-23T02:20:42.3841194Z 
2022-11-23T02:20:42.3841241Z OK
2022-11-23T02:20:42.3841245Z 
2022-11-23T02:20:42.3841326Z Generating XML reports...
2022-11-23T02:20:42.3841753Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021554.xml
2022-11-23T02:20:42.3842025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3842149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3842427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3842558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3842744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp874s0iu5
2022-11-23T02:20:42.3842936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp874s0iu5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3843179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3843184Z 
2022-11-23T02:20:42.3843257Z Running tests...
2022-11-23T02:20:42.3843451Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3843745Z   test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22310
2022-11-23T02:20:42.3843898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22311
2022-11-23T02:20:42.3844048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22312
2022-11-23T02:20:42.3844203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22313
2022-11-23T02:20:42.3844478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3844596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3844869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3845085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3845269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptro963wf
2022-11-23T02:20:42.3845541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3845662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3845940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3846073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3846257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit1cilk2
2022-11-23T02:20:42.3846438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptro963wf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3846664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit1cilk2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3847055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3847177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3847457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3847587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3847804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5twla8x_
2022-11-23T02:20:42.3847996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5twla8x_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3848265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3848376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3848652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3848785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3848963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8uijybn
2022-11-23T02:20:42.3849150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8uijybn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3849314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3849476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3849637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3849786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3849964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3850136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3850437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3850608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3850774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3851071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3851365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3851825Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3851879Z ok (2.774s)
2022-11-23T02:20:42.3851897Z 
2022-11-23T02:20:42.3852083Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3852154Z Ran 1 test in 2.774s
2022-11-23T02:20:42.3852159Z 
2022-11-23T02:20:42.3852215Z OK
2022-11-23T02:20:42.3852220Z 
2022-11-23T02:20:42.3852297Z Generating XML reports...
2022-11-23T02:20:42.3852727Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021558.xml
2022-11-23T02:20:42.3853001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3853123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3853443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3853566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3853752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtncfc8b
2022-11-23T02:20:42.3853947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtncfc8b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3854200Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3854206Z 
2022-11-23T02:20:42.3854273Z Running tests...
2022-11-23T02:20:42.3854495Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3854787Z   test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22557
2022-11-23T02:20:42.3855093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22558
2022-11-23T02:20:42.3855261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22559
2022-11-23T02:20:42.3855400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22560
2022-11-23T02:20:42.3855680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3855801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3856083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3856219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3856404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9fm0vw9
2022-11-23T02:20:42.3856597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9fm0vw9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3856868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3856978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3857335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3857468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3857648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt48lwaoy
2022-11-23T02:20:42.3857839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt48lwaoy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3858115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3858236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3858515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3858645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3858816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wrw7pps
2022-11-23T02:20:42.3859009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wrw7pps/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3859277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3859396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3859674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3859804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3859988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbt7p83z
2022-11-23T02:20:42.3860235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbt7p83z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3860453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3860615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3860775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3860933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3861034Z ok (2.688s)
2022-11-23T02:20:42.3861039Z 
2022-11-23T02:20:42.3861235Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3861309Z Ran 1 test in 2.688s
2022-11-23T02:20:42.3861314Z 
2022-11-23T02:20:42.3861371Z OK
2022-11-23T02:20:42.3861376Z 
2022-11-23T02:20:42.3861457Z Generating XML reports...
2022-11-23T02:20:42.3861872Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021603.xml
2022-11-23T02:20:42.3862144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3862266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3862542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3862673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3862859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj9n52vu
2022-11-23T02:20:42.3863050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj9n52vu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3863305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3863310Z 
2022-11-23T02:20:42.3863381Z Running tests...
2022-11-23T02:20:42.3863563Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3863846Z   test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22824
2022-11-23T02:20:42.3863999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22825
2022-11-23T02:20:42.3864146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22826
2022-11-23T02:20:42.3864296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22827
2022-11-23T02:20:42.3864570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3864690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3864970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3865091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3865271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02ocssfw
2022-11-23T02:20:42.3865462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02ocssfw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3865732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3865850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3866133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3866265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3866448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9l2zpev
2022-11-23T02:20:42.3866677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9l2zpev/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3866937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3867055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3867335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3867462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3867688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrih4d8w
2022-11-23T02:20:42.3867880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrih4d8w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3868147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3868270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3868535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3868667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3868844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglthh3j1
2022-11-23T02:20:42.3869035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglthh3j1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3869194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3869358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3869518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3869677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3869851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3870015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3870183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3870354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3870651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3870947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3871236Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3871524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3871589Z ok (3.000s)
2022-11-23T02:20:42.3871594Z 
2022-11-23T02:20:42.3871787Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3871849Z Ran 1 test in 3.000s
2022-11-23T02:20:42.3871854Z 
2022-11-23T02:20:42.3872029Z OK
2022-11-23T02:20:42.3872035Z 
2022-11-23T02:20:42.3872115Z Generating XML reports...
2022-11-23T02:20:42.3872544Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021608.xml
2022-11-23T02:20:42.3872818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3872940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3873250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3873384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3873569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2m01rre
2022-11-23T02:20:42.3873751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2m01rre/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3874004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3874010Z 
2022-11-23T02:20:42.3874081Z Running tests...
2022-11-23T02:20:42.3874273Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3874600Z   test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23071
2022-11-23T02:20:42.3874756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23072
2022-11-23T02:20:42.3874909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23073
2022-11-23T02:20:42.3875060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23074
2022-11-23T02:20:42.3875334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3875442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3875722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3875852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3876037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcman94m6
2022-11-23T02:20:42.3876304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3876421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3876695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3876828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3876998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r2f8vbj
2022-11-23T02:20:42.3877189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcman94m6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3877377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r2f8vbj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3877644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3877762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3878041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3878174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3878355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe34ez9u3
2022-11-23T02:20:42.3878545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe34ez9u3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3878900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3879021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3879303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3879432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3879610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_s_xxu5
2022-11-23T02:20:42.3879859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_s_xxu5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3880021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3880184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3880332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3880494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3880669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3880871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3881040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3881215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3881517Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3881812Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3882099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3882375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3882444Z ok (2.898s)
2022-11-23T02:20:42.3882449Z 
2022-11-23T02:20:42.3882641Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3882712Z Ran 1 test in 2.898s
2022-11-23T02:20:42.3882717Z 
2022-11-23T02:20:42.3882776Z OK
2022-11-23T02:20:42.3882781Z 
2022-11-23T02:20:42.3882862Z Generating XML reports...
2022-11-23T02:20:42.3883289Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021613.xml
2022-11-23T02:20:42.3883751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3883871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3884140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3884270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3884457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi8vwo_o
2022-11-23T02:20:42.3884652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi8vwo_o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3884970Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3884976Z 
2022-11-23T02:20:42.3885052Z Running tests...
2022-11-23T02:20:42.3885251Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3885547Z   test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23318
2022-11-23T02:20:42.3885701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23319
2022-11-23T02:20:42.3885841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23320
2022-11-23T02:20:42.3885994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23321
2022-11-23T02:20:42.3886267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3886388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3886772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3886907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3887094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi98kjy2f
2022-11-23T02:20:42.3887290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi98kjy2f/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3887553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3887714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3887994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3888126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3888307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5sn18v0
2022-11-23T02:20:42.3888499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5sn18v0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3888770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3888890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3889165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3889285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3889469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyn3mszs
2022-11-23T02:20:42.3889662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyn3mszs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3889998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3890119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3890397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3890530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3890709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhsb1u2o
2022-11-23T02:20:42.3890890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhsb1u2o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3891051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3891210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3891367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3891525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3891701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3891875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3892175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3892345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3892503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3892800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3893123Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3893411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3893471Z ok (2.779s)
2022-11-23T02:20:42.3893476Z 
2022-11-23T02:20:42.3893670Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3893743Z Ran 1 test in 2.779s
2022-11-23T02:20:42.3893748Z 
2022-11-23T02:20:42.3893805Z OK
2022-11-23T02:20:42.3893810Z 
2022-11-23T02:20:42.3893890Z Generating XML reports...
2022-11-23T02:20:42.3894303Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021618.xml
2022-11-23T02:20:42.3894602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3894721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3895106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3895240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3895426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfoykcmf
2022-11-23T02:20:42.3895623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfoykcmf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3895872Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3895877Z 
2022-11-23T02:20:42.3895947Z Running tests...
2022-11-23T02:20:42.3896130Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3896418Z   test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23565
2022-11-23T02:20:42.3896574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23566
2022-11-23T02:20:42.3896726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23567
2022-11-23T02:20:42.3896876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23568
2022-11-23T02:20:42.3897150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3897273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3897554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3897688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3897859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuahkwdjt
2022-11-23T02:20:42.3898057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuahkwdjt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3898324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3898446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3898722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3898853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3899037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8dx5ahp
2022-11-23T02:20:42.3899228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8dx5ahp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3899487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3899604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3899928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3900059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3900240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgjpgwtk
2022-11-23T02:20:42.3900437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgjpgwtk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3900707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3900864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3901254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3901375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3901555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42153qq2
2022-11-23T02:20:42.3901742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42153qq2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3901903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3902064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3902224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3902381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3902556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3902720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3902891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3903062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3903361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3903659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3903950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3904236Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3904302Z ok (2.777s)
2022-11-23T02:20:42.3904308Z 
2022-11-23T02:20:42.3904499Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3904561Z Ran 1 test in 2.777s
2022-11-23T02:20:42.3904575Z 
2022-11-23T02:20:42.3904624Z OK
2022-11-23T02:20:42.3904630Z 
2022-11-23T02:20:42.3904711Z Generating XML reports...
2022-11-23T02:20:42.3905135Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021622.xml
2022-11-23T02:20:42.3905405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3905523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3905801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3905933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3906116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy570mjxr
2022-11-23T02:20:42.3906298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy570mjxr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3906582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3906588Z 
2022-11-23T02:20:42.3906660Z Running tests...
2022-11-23T02:20:42.3906853Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3907155Z   test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23812
2022-11-23T02:20:42.3907309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23813
2022-11-23T02:20:42.3907488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23814
2022-11-23T02:20:42.3907638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23815
2022-11-23T02:20:42.3907909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3908022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3908299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3908434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3908619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwb5lp17
2022-11-23T02:20:42.3908810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwb5lp17/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3909079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3909200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3909477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3909598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3909783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80wcr95l
2022-11-23T02:20:42.3909973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80wcr95l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3910243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3910364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3910641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3910774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3910955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnx6vt_u
2022-11-23T02:20:42.3911148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnx6vt_u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3911409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3911528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3911805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3911934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3912115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7scspw3l
2022-11-23T02:20:42.3912307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7scspw3l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3912467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3912624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3912801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3913065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3913240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3913415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3913717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3913888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3914099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3914396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3914691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3914979Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3915033Z ok (2.677s)
2022-11-23T02:20:42.3915038Z 
2022-11-23T02:20:42.3915233Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3915307Z Ran 1 test in 2.678s
2022-11-23T02:20:42.3915311Z 
2022-11-23T02:20:42.3915370Z OK
2022-11-23T02:20:42.3915374Z 
2022-11-23T02:20:42.3915454Z Generating XML reports...
2022-11-23T02:20:42.3915884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021627.xml
2022-11-23T02:20:42.3916155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3916275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3916544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3916678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3916863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5jat3dk
2022-11-23T02:20:42.3917058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5jat3dk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3917308Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3917315Z 
2022-11-23T02:20:42.3917386Z Running tests...
2022-11-23T02:20:42.3917584Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3917883Z   test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24059
2022-11-23T02:20:42.3918105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24060
2022-11-23T02:20:42.3918247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24061
2022-11-23T02:20:42.3918399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24062
2022-11-23T02:20:42.3918677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3918798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3919080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3919213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3919392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3nmafo6
2022-11-23T02:20:42.3919615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3nmafo6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3919887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3919996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3920274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3920406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3920590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxt6can_
2022-11-23T02:20:42.3920811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxt6can_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3921080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3921200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3921479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3921599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3921845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmo5dj5a
2022-11-23T02:20:42.3922040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmo5dj5a/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3922312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3922432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3922710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3922839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3923020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvfw77qg
2022-11-23T02:20:42.3923212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvfw77qg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3923363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3923523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3923681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3923843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3924016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3924189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3924485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3924655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3924823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3925197Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3925494Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3925784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3925849Z ok (2.774s)
2022-11-23T02:20:42.3925854Z 
2022-11-23T02:20:42.3926049Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3926157Z Ran 1 test in 2.775s
2022-11-23T02:20:42.3926163Z 
2022-11-23T02:20:42.3926221Z OK
2022-11-23T02:20:42.3926225Z 
2022-11-23T02:20:42.3926322Z Generating XML reports...
2022-11-23T02:20:42.3926736Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021631.xml
2022-11-23T02:20:42.3927008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3927129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3927437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3927570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3927756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueiyyknl
2022-11-23T02:20:42.3927950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueiyyknl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3928233Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3928238Z 
2022-11-23T02:20:42.3928305Z Running tests...
2022-11-23T02:20:42.3928487Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3928773Z   test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24306
2022-11-23T02:20:42.3928927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24307
2022-11-23T02:20:42.3929081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24308
2022-11-23T02:20:42.3929230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24309
2022-11-23T02:20:42.3929506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3929624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3929903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3930034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3930206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0c03izt
2022-11-23T02:20:42.3930398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0c03izt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3930669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3930788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3931063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3931193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3931375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2y25d7_
2022-11-23T02:20:42.3931563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2y25d7_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3931821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3931945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3932221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3932354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3932653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiipk70t0
2022-11-23T02:20:42.3932878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiipk70t0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3933152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3933273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3933549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3933668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3933849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinmje6zz
2022-11-23T02:20:42.3934068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinmje6zz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3934233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3934392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3934553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3934712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3934997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3935216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3935377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3935550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3935854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3936151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3936444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3936733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3936798Z ok (2.777s)
2022-11-23T02:20:42.3936804Z 
2022-11-23T02:20:42.3936999Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3937060Z Ran 1 test in 2.777s
2022-11-23T02:20:42.3937080Z 
2022-11-23T02:20:42.3937129Z OK
2022-11-23T02:20:42.3937134Z 
2022-11-23T02:20:42.3937215Z Generating XML reports...
2022-11-23T02:20:42.3937638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021636.xml
2022-11-23T02:20:42.3937909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3938032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3938309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3938442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3938626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9klkwp58
2022-11-23T02:20:42.3938806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9klkwp58/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3939058Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3939063Z 
2022-11-23T02:20:42.3939132Z Running tests...
2022-11-23T02:20:42.3939320Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3939666Z   test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24553
2022-11-23T02:20:42.3939821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24554
2022-11-23T02:20:42.3939975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24555
2022-11-23T02:20:42.3940125Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24556
2022-11-23T02:20:42.3940401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3940509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3940827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3940957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3941139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8ws24dm
2022-11-23T02:20:42.3941333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8ws24dm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3941601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3941715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3941992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3942122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3942302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2sr2jch
2022-11-23T02:20:42.3942489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2sr2jch/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3942753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3942873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3943146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3943265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3943444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2u0oyyb
2022-11-23T02:20:42.3943634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2u0oyyb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3944021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3944140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3944417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3944548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3944729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uwspims
2022-11-23T02:20:42.3944919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uwspims/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3945070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3945229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3945385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3945544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3945714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3945887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3946220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3946390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3946559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3946840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3947131Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3947444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3947509Z ok (2.880s)
2022-11-23T02:20:42.3947514Z 
2022-11-23T02:20:42.3947712Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3947784Z Ran 1 test in 2.880s
2022-11-23T02:20:42.3947789Z 
2022-11-23T02:20:42.3947848Z OK
2022-11-23T02:20:42.3947852Z 
2022-11-23T02:20:42.3947933Z Generating XML reports...
2022-11-23T02:20:42.3948558Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021641.xml
2022-11-23T02:20:42.3948832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3948951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3949228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3949357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3949539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohka41b1
2022-11-23T02:20:42.3949736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohka41b1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3949987Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3949992Z 
2022-11-23T02:20:42.3950060Z Running tests...
2022-11-23T02:20:42.3950241Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3950521Z   test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24800
2022-11-23T02:20:42.3950676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24801
2022-11-23T02:20:42.3950826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24802
2022-11-23T02:20:42.3950970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24803
2022-11-23T02:20:42.3951245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3951364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3951639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3951773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3951946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaaw3218
2022-11-23T02:20:42.3952136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaaw3218/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3952403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3952523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3952830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3952964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3953146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuzzqj0l
2022-11-23T02:20:42.3953336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuzzqj0l/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3953594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3953713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3954016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3954142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3954322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl8jrn0i
2022-11-23T02:20:42.3954582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl8jrn0i/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3954852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3954970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3955245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3955366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3955548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xntoqip
2022-11-23T02:20:42.3955739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xntoqip/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3955898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3956057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3956215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3956373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3956436Z ok (2.578s)
2022-11-23T02:20:42.3956441Z 
2022-11-23T02:20:42.3956623Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3956694Z Ran 1 test in 2.578s
2022-11-23T02:20:42.3956700Z 
2022-11-23T02:20:42.3956755Z OK
2022-11-23T02:20:42.3956762Z 
2022-11-23T02:20:42.3956839Z Generating XML reports...
2022-11-23T02:20:42.3957262Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021645.xml
2022-11-23T02:20:42.3957528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3957648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3957925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3958055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3958227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9a7x3boa
2022-11-23T02:20:42.3958416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9a7x3boa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3958666Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3958673Z 
2022-11-23T02:20:42.3958741Z Running tests...
2022-11-23T02:20:42.3958934Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3959248Z   test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25055
2022-11-23T02:20:42.3959407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25056
2022-11-23T02:20:42.3959554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25057
2022-11-23T02:20:42.3959703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25058
2022-11-23T02:20:42.3959966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3960083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3960408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3960534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3960713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu518dah
2022-11-23T02:20:42.3960904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu518dah/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3961173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3961291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3961561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3961692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3961937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4ft0jyv
2022-11-23T02:20:42.3962128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4ft0jyv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3962391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3962511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3962788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3962918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3963097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtnjn_yu
2022-11-23T02:20:42.3963276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtnjn_yu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3963541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3963661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3963932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3964062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3964244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9oyhc1jp
2022-11-23T02:20:42.3964434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9oyhc1jp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3964593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3964743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3964963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3965127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3965190Z ok (3.676s)
2022-11-23T02:20:42.3965195Z 
2022-11-23T02:20:42.3965387Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3965458Z Ran 1 test in 3.676s
2022-11-23T02:20:42.3965463Z 
2022-11-23T02:20:42.3965520Z OK
2022-11-23T02:20:42.3965559Z 
2022-11-23T02:20:42.3965640Z Generating XML reports...
2022-11-23T02:20:42.3966066Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021650.xml
2022-11-23T02:20:42.3966330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3966447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3966721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3966881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3967058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1zgmo5w
2022-11-23T02:20:42.3967245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1zgmo5w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3967494Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3967499Z 
2022-11-23T02:20:42.3967568Z Running tests...
2022-11-23T02:20:42.3967749Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3968046Z   test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25310
2022-11-23T02:20:42.3968198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25311
2022-11-23T02:20:42.3968351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25312
2022-11-23T02:20:42.3968499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25313
2022-11-23T02:20:42.3968767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3968888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3969166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3969296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3969468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2tsyy0r
2022-11-23T02:20:42.3969658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2tsyy0r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3969927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3970047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3970324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3970454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3970636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcffsq6hl
2022-11-23T02:20:42.3970828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcffsq6hl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3971097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3971205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3971478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3971615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3971792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qsx0st7
2022-11-23T02:20:42.3971982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qsx0st7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3972280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3972401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3972673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3972793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3972975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg335p5uz
2022-11-23T02:20:42.3973284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg335p5uz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3973478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3973635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3973796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3973952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3974015Z ok (2.574s)
2022-11-23T02:20:42.3974020Z 
2022-11-23T02:20:42.3974216Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3974278Z Ran 1 test in 2.574s
2022-11-23T02:20:42.3974283Z 
2022-11-23T02:20:42.3974344Z OK
2022-11-23T02:20:42.3974350Z 
2022-11-23T02:20:42.3974431Z Generating XML reports...
2022-11-23T02:20:42.3974936Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021655.xml
2022-11-23T02:20:42.3975213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3975333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3975614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3975745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3975916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavqceoz1
2022-11-23T02:20:42.3976107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavqceoz1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3976356Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3976362Z 
2022-11-23T02:20:42.3976429Z Running tests...
2022-11-23T02:20:42.3976619Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3976926Z   test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25565
2022-11-23T02:20:42.3977082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25566
2022-11-23T02:20:42.3977234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25567
2022-11-23T02:20:42.3977385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25568
2022-11-23T02:20:42.3977647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3977764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3978036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3978167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3978349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pkm90xk
2022-11-23T02:20:42.3978542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pkm90xk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3978858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3978976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3979249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3979370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3979550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwznlg8r
2022-11-23T02:20:42.3979808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwznlg8r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3980074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3980193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3980472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3980601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3980780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59f1_wdt
2022-11-23T02:20:42.3980959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59f1_wdt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3981222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3981340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3981611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3981739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3981921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtatded6
2022-11-23T02:20:42.3982110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtatded6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3982269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3982425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3982573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3982732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3983015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3983189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3983360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3983532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3983832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3984125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3984412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3984687Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3984752Z ok (2.678s)
2022-11-23T02:20:42.3984757Z 
2022-11-23T02:20:42.3984949Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3985024Z Ran 1 test in 2.678s
2022-11-23T02:20:42.3985030Z 
2022-11-23T02:20:42.3985088Z OK
2022-11-23T02:20:42.3985124Z 
2022-11-23T02:20:42.3985208Z Generating XML reports...
2022-11-23T02:20:42.3985629Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021700.xml
2022-11-23T02:20:42.3985898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3986016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3986284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3986443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3986625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_mkwew9
2022-11-23T02:20:42.3986812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_mkwew9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3987066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3987071Z 
2022-11-23T02:20:42.3987139Z Running tests...
2022-11-23T02:20:42.3987330Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3987630Z   test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25812
2022-11-23T02:20:42.3987779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25813
2022-11-23T02:20:42.3987922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25814
2022-11-23T02:20:42.3988071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25815
2022-11-23T02:20:42.3988344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3988466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3988747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3988877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3989132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zmf4skp
2022-11-23T02:20:42.3989323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zmf4skp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3989585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3989706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3989980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3990110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3990293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_vx87b_
2022-11-23T02:20:42.3990483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_vx87b_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3990749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3990868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3991145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3991267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3991447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga3lsnes
2022-11-23T02:20:42.3991636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga3lsnes/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3991932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3992052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3992330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3992459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3992703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprd56z962
2022-11-23T02:20:42.3992884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprd56z962/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3993078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.3993238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.3993394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.3993548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.3993722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.3993895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.3994064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.3994233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.3994523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3994815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3995105Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3995389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.3995452Z ok (2.774s)
2022-11-23T02:20:42.3995457Z 
2022-11-23T02:20:42.3995649Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3995720Z Ran 1 test in 2.774s
2022-11-23T02:20:42.3995725Z 
2022-11-23T02:20:42.3995781Z OK
2022-11-23T02:20:42.3995785Z 
2022-11-23T02:20:42.3995861Z Generating XML reports...
2022-11-23T02:20:42.3996274Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021704.xml
2022-11-23T02:20:42.3996549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3996671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3996949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3997079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3997260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9lzmn0k
2022-11-23T02:20:42.3997450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9lzmn0k/_remote_module_non_scriptable.py
2022-11-23T02:20:42.3997699Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.3997706Z 
2022-11-23T02:20:42.3997772Z Running tests...
2022-11-23T02:20:42.3997951Z ----------------------------------------------------------------------
2022-11-23T02:20:42.3998233Z   test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26059
2022-11-23T02:20:42.3998416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26060
2022-11-23T02:20:42.3998571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26061
2022-11-23T02:20:42.3998720Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26062
2022-11-23T02:20:42.3998989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.3999106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.3999384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.3999532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.3999710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i0q5bog
2022-11-23T02:20:42.3999902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i0q5bog/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4000171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4000289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4000565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4000692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4000869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mc9gw5x
2022-11-23T02:20:42.4001059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mc9gw5x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4001316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4001433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4001710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4001839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4002016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz2z6i92
2022-11-23T02:20:42.4002205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz2z6i92/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4002472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4002591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4002857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4002986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4003168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuppc3qi
2022-11-23T02:20:42.4003358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuppc3qi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4003516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4003675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4003831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4003991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4004054Z ok (2.578s)
2022-11-23T02:20:42.4004059Z 
2022-11-23T02:20:42.4004240Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4004313Z Ran 1 test in 2.578s
2022-11-23T02:20:42.4004318Z 
2022-11-23T02:20:42.4004373Z OK
2022-11-23T02:20:42.4004378Z 
2022-11-23T02:20:42.4004484Z Generating XML reports...
2022-11-23T02:20:42.4004987Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021709.xml
2022-11-23T02:20:42.4005260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4005380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4005654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4005833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4006004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzaiy7ccy
2022-11-23T02:20:42.4006195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzaiy7ccy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4006573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4006579Z 
2022-11-23T02:20:42.4006648Z Running tests...
2022-11-23T02:20:42.4006838Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4007117Z   test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26306
2022-11-23T02:20:42.4007273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26307
2022-11-23T02:20:42.4007424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26308
2022-11-23T02:20:42.4007566Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26309
2022-11-23T02:20:42.4007838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4007953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4008236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4008367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4008548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrm6jtew
2022-11-23T02:20:42.4008742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrm6jtew/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4009017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4009138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4009407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4009540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4009720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8232s34e
2022-11-23T02:20:42.4009976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8232s34e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4010245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4010364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4010640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4010773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4010946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpundn9wg_
2022-11-23T02:20:42.4011134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpundn9wg_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4011432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4011553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4011830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4011960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4012142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnv45467
2022-11-23T02:20:42.4012332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnv45467/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4012522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4012672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4012892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4013050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4013112Z ok (2.781s)
2022-11-23T02:20:42.4013118Z 
2022-11-23T02:20:42.4013307Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4013378Z Ran 1 test in 2.781s
2022-11-23T02:20:42.4013383Z 
2022-11-23T02:20:42.4013439Z OK
2022-11-23T02:20:42.4013444Z 
2022-11-23T02:20:42.4013522Z Generating XML reports...
2022-11-23T02:20:42.4013933Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021713.xml
2022-11-23T02:20:42.4014205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4014324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4014600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4014734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4015042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy07k6m3g
2022-11-23T02:20:42.4015257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy07k6m3g/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4015510Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4015515Z 
2022-11-23T02:20:42.4015584Z Running tests...
2022-11-23T02:20:42.4015774Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4016048Z   test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26553
2022-11-23T02:20:42.4016198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26554
2022-11-23T02:20:42.4016349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26555
2022-11-23T02:20:42.4016498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26556
2022-11-23T02:20:42.4016768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4016887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4017167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4017296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4017470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsxsesu0
2022-11-23T02:20:42.4017663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsxsesu0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4017988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4018109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4018386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4018516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4018696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ymj63fa
2022-11-23T02:20:42.4018885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ymj63fa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4019177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4019296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4019572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4019703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4019881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i5xss6s
2022-11-23T02:20:42.4020070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i5xss6s/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4020336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4020456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4020731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4020976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4021158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1458thca
2022-11-23T02:20:42.4021346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1458thca/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4021503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4021659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4021816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4021974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4022036Z ok (2.775s)
2022-11-23T02:20:42.4022044Z 
2022-11-23T02:20:42.4022227Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4022297Z Ran 1 test in 2.776s
2022-11-23T02:20:42.4022302Z 
2022-11-23T02:20:42.4022356Z OK
2022-11-23T02:20:42.4022361Z 
2022-11-23T02:20:42.4022438Z Generating XML reports...
2022-11-23T02:20:42.4022860Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021718.xml
2022-11-23T02:20:42.4023132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4023252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4023526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4023657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4023829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95xsbvhp
2022-11-23T02:20:42.4024025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95xsbvhp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4024274Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4024279Z 
2022-11-23T02:20:42.4024348Z Running tests...
2022-11-23T02:20:42.4024569Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4024831Z   test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26820
2022-11-23T02:20:42.4024984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26821
2022-11-23T02:20:42.4025131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26822
2022-11-23T02:20:42.4025278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26823
2022-11-23T02:20:42.4025568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4025687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4025963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4026097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4026279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27muhwxw
2022-11-23T02:20:42.4026469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27muhwxw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4026735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4026854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4027120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4027253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4027433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7134pex
2022-11-23T02:20:42.4027625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7134pex/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4027888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4028006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4028280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4028410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4028589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputkmpxmb
2022-11-23T02:20:42.4028773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputkmpxmb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4029040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4029158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4029432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4029564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4029743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrb7tbb2
2022-11-23T02:20:42.4029928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrb7tbb2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4030089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4030241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4030396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4030555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4030618Z ok (2.476s)
2022-11-23T02:20:42.4030622Z 
2022-11-23T02:20:42.4030956Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4031029Z Ran 1 test in 2.476s
2022-11-23T02:20:42.4031034Z 
2022-11-23T02:20:42.4031092Z OK
2022-11-23T02:20:42.4031097Z 
2022-11-23T02:20:42.4031177Z Generating XML reports...
2022-11-23T02:20:42.4031566Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021723.xml
2022-11-23T02:20:42.4031827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4031975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4032251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4032379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4032560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_k47lal6
2022-11-23T02:20:42.4032748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_k47lal6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4032998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4033003Z 
2022-11-23T02:20:42.4033071Z Running tests...
2022-11-23T02:20:42.4033254Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4033513Z   test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27055
2022-11-23T02:20:42.4033667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27056
2022-11-23T02:20:42.4033816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27057
2022-11-23T02:20:42.4033964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27058
2022-11-23T02:20:42.4034235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4034355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4034631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4034759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4034929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3f1f4rfg
2022-11-23T02:20:42.4035361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4035480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4035757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4035889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4036069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h9k8lfz
2022-11-23T02:20:42.4036260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3f1f4rfg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4036447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h9k8lfz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4036715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4036827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4037101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4037232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4037442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9le970n9
2022-11-23T02:20:42.4037631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9le970n9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4037896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4038014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4038289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4038408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4038618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuzncvp7
2022-11-23T02:20:42.4038806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuzncvp7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4038965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4039127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4039353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4039511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4039574Z ok (2.475s)
2022-11-23T02:20:42.4039580Z 
2022-11-23T02:20:42.4039774Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4039836Z Ran 1 test in 2.475s
2022-11-23T02:20:42.4039843Z 
2022-11-23T02:20:42.4039897Z OK
2022-11-23T02:20:42.4039902Z 
2022-11-23T02:20:42.4039981Z Generating XML reports...
2022-11-23T02:20:42.4040370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021727.xml
2022-11-23T02:20:42.4040645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4040767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4041044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4041181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4041352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ynqbg2z
2022-11-23T02:20:42.4041544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ynqbg2z/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4041800Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4041805Z 
2022-11-23T02:20:42.4041872Z Running tests...
2022-11-23T02:20:42.4042063Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4042333Z   test_dynamic_and_static_init_rpc_together (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27290
2022-11-23T02:20:42.4042486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27291
2022-11-23T02:20:42.4042634Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27292
2022-11-23T02:20:42.4042784Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27293
2022-11-23T02:20:42.4043045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4043164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4043511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4043640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4043851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hit_fv8
2022-11-23T02:20:42.4044045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hit_fv8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4044323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4044441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4044710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4044841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4045135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0lifhx9
2022-11-23T02:20:42.4045325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0lifhx9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4045597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4045720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4045997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4046125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4046302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h3a8t5w
2022-11-23T02:20:42.4046481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h3a8t5w/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4046752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4046869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4047148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4047282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4047461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3flbp9mm
2022-11-23T02:20:42.4047650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3flbp9mm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4047807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4047966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4048115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4048273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4048451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4048627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4048802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.4049097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4049390Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4049561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.4049858Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4050140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4050446Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4050715Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4050781Z ok (2.675s)
2022-11-23T02:20:42.4050786Z 
2022-11-23T02:20:42.4050980Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4051050Z Ran 1 test in 2.676s
2022-11-23T02:20:42.4051055Z 
2022-11-23T02:20:42.4051112Z OK
2022-11-23T02:20:42.4051155Z 
2022-11-23T02:20:42.4051234Z Generating XML reports...
2022-11-23T02:20:42.4051624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021731.xml
2022-11-23T02:20:42.4051887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4052011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4052289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4052421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4052602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah52cftb
2022-11-23T02:20:42.4052792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah52cftb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4053043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4053050Z 
2022-11-23T02:20:42.4053118Z Running tests...
2022-11-23T02:20:42.4053312Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4054085Z   test_dynamic_rpc_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84116 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.256s)
2022-11-23T02:20:42.4054100Z 
2022-11-23T02:20:42.4054393Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4054465Z Ran 1 test in 0.256s
2022-11-23T02:20:42.4054470Z 
2022-11-23T02:20:42.4054536Z OK (skipped=1)
2022-11-23T02:20:42.4054540Z 
2022-11-23T02:20:42.4054622Z Generating XML reports...
2022-11-23T02:20:42.4055095Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021736.xml
2022-11-23T02:20:42.4055366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4055488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4055765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4055900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4056075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylbhxkx3
2022-11-23T02:20:42.4056267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylbhxkx3/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4056515Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4056523Z 
2022-11-23T02:20:42.4056591Z Running tests...
2022-11-23T02:20:42.4056782Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4057127Z   test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27510
2022-11-23T02:20:42.4057286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27511
2022-11-23T02:20:42.4057437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27512
2022-11-23T02:20:42.4057587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27513
2022-11-23T02:20:42.4057851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4057969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4058358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4058488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4058669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiho2lpsf
2022-11-23T02:20:42.4058864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiho2lpsf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4059134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4059254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4059518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4059650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4059832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfh8kb6i5
2022-11-23T02:20:42.4060025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfh8kb6i5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4060292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4060411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4060691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4060822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4061003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0qhp8we
2022-11-23T02:20:42.4061184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0qhp8we/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4061450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4061571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4061845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4061973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4062153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp4stmz5
2022-11-23T02:20:42.4062345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp4stmz5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4062505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4062730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4062881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4063040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4063140Z skip: Need at least 2 CUDA devices (2.176s)
2022-11-23T02:20:42.4063145Z 
2022-11-23T02:20:42.4063338Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4063409Z Ran 1 test in 2.176s
2022-11-23T02:20:42.4063414Z 
2022-11-23T02:20:42.4063518Z OK (skipped=1)
2022-11-23T02:20:42.4063524Z 
2022-11-23T02:20:42.4063606Z Generating XML reports...
2022-11-23T02:20:42.4063991Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021738.xml
2022-11-23T02:20:42.4064251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4064370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4064643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4064801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4064982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_1x4ehh
2022-11-23T02:20:42.4065168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_1x4ehh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4065420Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4065425Z 
2022-11-23T02:20:42.4065492Z Running tests...
2022-11-23T02:20:42.4065682Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4065925Z   test_dynamic_rpc_init_rpc (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27565
2022-11-23T02:20:42.4066075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27566
2022-11-23T02:20:42.4066228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27567
2022-11-23T02:20:42.4066377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27568
2022-11-23T02:20:42.4066644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4066765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4067043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4067174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4067345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv5rx0ys
2022-11-23T02:20:42.4067535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv5rx0ys/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4067803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4067923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4068195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4068326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4068508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v99yspt
2022-11-23T02:20:42.4068697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v99yspt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4068964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4069074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4069353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4069486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4069660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlfdbalc
2022-11-23T02:20:42.4069851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlfdbalc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4070147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4070264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4070537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4070669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4070838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcctk7iu6
2022-11-23T02:20:42.4071028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcctk7iu6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4071220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4071377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4071535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4071691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4071754Z ok (2.579s)
2022-11-23T02:20:42.4071759Z 
2022-11-23T02:20:42.4071952Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4072012Z Ran 1 test in 2.579s
2022-11-23T02:20:42.4072017Z 
2022-11-23T02:20:42.4072073Z OK
2022-11-23T02:20:42.4072078Z 
2022-11-23T02:20:42.4072155Z Generating XML reports...
2022-11-23T02:20:42.4072542Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021742.xml
2022-11-23T02:20:42.4072814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4072934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4073213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4073342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4073523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiakeojsk
2022-11-23T02:20:42.4073704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiakeojsk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4074068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4074074Z 
2022-11-23T02:20:42.4074140Z Running tests...
2022-11-23T02:20:42.4074337Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4074601Z   test_dynamic_rpc_init_rpc_without_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27776
2022-11-23T02:20:42.4074753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27777
2022-11-23T02:20:42.4074908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27778
2022-11-23T02:20:42.4075057Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27779
2022-11-23T02:20:42.4075319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4075439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4075713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4075848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4076027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32wdn_5_
2022-11-23T02:20:42.4076220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32wdn_5_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4076517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4076639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4076917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4077039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4077216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22iy3btc
2022-11-23T02:20:42.4077407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22iy3btc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4077703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4077821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4078167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4078297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4078475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fd9czk0
2022-11-23T02:20:42.4078656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fd9czk0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4078922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4079039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4079316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4079447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4079626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzs25un4
2022-11-23T02:20:42.4079817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzs25un4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4079976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4080134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4080282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4080439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4080504Z ok (2.180s)
2022-11-23T02:20:42.4080509Z 
2022-11-23T02:20:42.4080703Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4080774Z Ran 1 test in 2.180s
2022-11-23T02:20:42.4080779Z 
2022-11-23T02:20:42.4080834Z OK
2022-11-23T02:20:42.4080839Z 
2022-11-23T02:20:42.4080918Z Generating XML reports...
2022-11-23T02:20:42.4081309Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021746.xml
2022-11-23T02:20:42.4081581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4081690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4081969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4082100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4082282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55ak5lr2
2022-11-23T02:20:42.4082471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55ak5lr2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4082722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4082727Z 
2022-11-23T02:20:42.4082857Z Running tests...
2022-11-23T02:20:42.4083051Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4083335Z   test_dynamic_rpc_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27835
2022-11-23T02:20:42.4083492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27836
2022-11-23T02:20:42.4083710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27837
2022-11-23T02:20:42.4083860Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27838
2022-11-23T02:20:42.4084167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4084287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4084569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4084700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4084884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpluy3nado
2022-11-23T02:20:42.4085133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpluy3nado/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4085406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4085525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4085801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4085932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4086112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1uf5kec
2022-11-23T02:20:42.4086305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1uf5kec/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4086575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4086683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4086963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4087160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4087342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1bpn_yk
2022-11-23T02:20:42.4087532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1bpn_yk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4087800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4087922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4088197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4088328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4088497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdt2zvvki
2022-11-23T02:20:42.4088689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdt2zvvki/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4088848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4089011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4089169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4089325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4089546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4089841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4090013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4090171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.4090338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.4090669Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4090962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4091254Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4091521Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4091784Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4092045Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4092306Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4092564Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4092818Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4092872Z ok (2.776s)
2022-11-23T02:20:42.4092878Z 
2022-11-23T02:20:42.4093074Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4093148Z Ran 1 test in 2.776s
2022-11-23T02:20:42.4093153Z 
2022-11-23T02:20:42.4093210Z OK
2022-11-23T02:20:42.4093214Z 
2022-11-23T02:20:42.4093295Z Generating XML reports...
2022-11-23T02:20:42.4093683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021750.xml
2022-11-23T02:20:42.4093957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4094079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4094355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4094476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4094659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63lkw7ze
2022-11-23T02:20:42.4094942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63lkw7ze/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4095262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4095267Z 
2022-11-23T02:20:42.4095335Z Running tests...
2022-11-23T02:20:42.4095529Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4095841Z   test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28058
2022-11-23T02:20:42.4095998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28059
2022-11-23T02:20:42.4096139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28060
2022-11-23T02:20:42.4096289Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28061
2022-11-23T02:20:42.4096563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4096733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4097010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4097141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4097326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8egom9a1
2022-11-23T02:20:42.4097519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8egom9a1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4097871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4097981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4098261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4098392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4098574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfl4sd11
2022-11-23T02:20:42.4098763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfl4sd11/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4099033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4099149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4099426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4099556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4099727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3w_2ln0
2022-11-23T02:20:42.4099912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3w_2ln0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4100182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4100302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4100576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4100708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4100888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjbnjsjg
2022-11-23T02:20:42.4101080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjbnjsjg/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4101233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4101393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4101554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4101711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4102008Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4102343Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4102601Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4102666Z ok (2.774s)
2022-11-23T02:20:42.4102671Z 
2022-11-23T02:20:42.4102867Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4102970Z Ran 1 test in 2.774s
2022-11-23T02:20:42.4102975Z 
2022-11-23T02:20:42.4103022Z OK
2022-11-23T02:20:42.4103027Z 
2022-11-23T02:20:42.4103106Z Generating XML reports...
2022-11-23T02:20:42.4103497Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021755.xml
2022-11-23T02:20:42.4103770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4103890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4104166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4104300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4104482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_f2_8k_
2022-11-23T02:20:42.4104664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_f2_8k_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4104914Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4104919Z 
2022-11-23T02:20:42.4104986Z Running tests...
2022-11-23T02:20:42.4105178Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4105440Z   test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28281
2022-11-23T02:20:42.4105591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28282
2022-11-23T02:20:42.4105742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28283
2022-11-23T02:20:42.4105892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28284
2022-11-23T02:20:42.4106162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4106273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4106550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4106682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4106929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp379k13oz
2022-11-23T02:20:42.4107121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp379k13oz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4107392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4107511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4107788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4107922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4108092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6asa8cz
2022-11-23T02:20:42.4108284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6asa8cz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4108585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4108707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4108983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4109112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4109290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnuj__bwj
2022-11-23T02:20:42.4109477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnuj__bwj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4109766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4109883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4110159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4110290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4110470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpevz4f2
2022-11-23T02:20:42.4110663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpevz4f2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4110822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4110980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4111138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4111286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4111350Z ok (2.174s)
2022-11-23T02:20:42.4111355Z 
2022-11-23T02:20:42.4111550Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4111623Z Ran 1 test in 2.174s
2022-11-23T02:20:42.4111629Z 
2022-11-23T02:20:42.4111687Z OK
2022-11-23T02:20:42.4111692Z 
2022-11-23T02:20:42.4111770Z Generating XML reports...
2022-11-23T02:20:42.4112159Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021800.xml
2022-11-23T02:20:42.4112427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4112537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4112813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4112944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4113124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsa57rqb
2022-11-23T02:20:42.4113321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsa57rqb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4113571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4113577Z 
2022-11-23T02:20:42.4113647Z Running tests...
2022-11-23T02:20:42.4113834Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4114097Z   test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28336
2022-11-23T02:20:42.4114242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28337
2022-11-23T02:20:42.4114393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28338
2022-11-23T02:20:42.4114541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28339
2022-11-23T02:20:42.4114838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4114957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4115234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4115362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4115544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoofq71o5
2022-11-23T02:20:42.4115734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoofq71o5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4116025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4116144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4116425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4116555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4116735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfqhx1h9
2022-11-23T02:20:42.4116922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfqhx1h9/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4117190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4117311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4117579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4117707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4118012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz8njssr
2022-11-23T02:20:42.4118205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz8njssr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4118472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4118590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4118865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4118995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4119174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kkdr8m6
2022-11-23T02:20:42.4119353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kkdr8m6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4119514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4119676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4119834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4119991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4120055Z ok (2.678s)
2022-11-23T02:20:42.4120061Z 
2022-11-23T02:20:42.4120254Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4120325Z Ran 1 test in 2.679s
2022-11-23T02:20:42.4120330Z 
2022-11-23T02:20:42.4120376Z OK
2022-11-23T02:20:42.4120381Z 
2022-11-23T02:20:42.4120463Z Generating XML reports...
2022-11-23T02:20:42.4120842Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021804.xml
2022-11-23T02:20:42.4121114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4121267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4121552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4121683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4121933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivz2eyij
2022-11-23T02:20:42.4122127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivz2eyij/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4122369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4122404Z 
2022-11-23T02:20:42.4122473Z Running tests...
2022-11-23T02:20:42.4122665Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4122929Z   test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28571
2022-11-23T02:20:42.4123082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28572
2022-11-23T02:20:42.4123232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28573
2022-11-23T02:20:42.4123382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28574
2022-11-23T02:20:42.4123653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4123762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4124043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4124173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4124356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi31m32x4
2022-11-23T02:20:42.4124549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi31m32x4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4124820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4125016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4125298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4125499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4125671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyw_atxp
2022-11-23T02:20:42.4125864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyw_atxp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4126133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4126253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4126528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4126658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4126838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65h9yarv
2022-11-23T02:20:42.4127027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65h9yarv/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4127294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4127404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4127682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4127813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4128030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dyf7qzb
2022-11-23T02:20:42.4128223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dyf7qzb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4128382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4128540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4128699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4128874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4128939Z ok (2.473s)
2022-11-23T02:20:42.4128945Z 
2022-11-23T02:20:42.4129137Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4129208Z Ran 1 test in 2.473s
2022-11-23T02:20:42.4129213Z 
2022-11-23T02:20:42.4129271Z OK
2022-11-23T02:20:42.4129275Z 
2022-11-23T02:20:42.4129357Z Generating XML reports...
2022-11-23T02:20:42.4129746Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021808.xml
2022-11-23T02:20:42.4130014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4130135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4130401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4130533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4130714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej5p5lh2
2022-11-23T02:20:42.4130904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej5p5lh2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4131155Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4131160Z 
2022-11-23T02:20:42.4131229Z Running tests...
2022-11-23T02:20:42.4131420Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4131671Z   test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28806
2022-11-23T02:20:42.4131814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28807
2022-11-23T02:20:42.4131965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28808
2022-11-23T02:20:42.4132114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28809
2022-11-23T02:20:42.4132384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4132502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4132783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4132915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4133095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnw5vzrm
2022-11-23T02:20:42.4133286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnw5vzrm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4133547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4133667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4133943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4134073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4134281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlhmnt55
2022-11-23T02:20:42.4134470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlhmnt55/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4134737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4134940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4135206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4135335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4135567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdhn7y71
2022-11-23T02:20:42.4135757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdhn7y71/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4136149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4136270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4136550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4136678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4136859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt420b1ij
2022-11-23T02:20:42.4137036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt420b1ij/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4137198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4137358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4137514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4137672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4137733Z ok (2.977s)
2022-11-23T02:20:42.4137738Z 
2022-11-23T02:20:42.4137935Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4138005Z Ran 1 test in 2.977s
2022-11-23T02:20:42.4138010Z 
2022-11-23T02:20:42.4138057Z OK
2022-11-23T02:20:42.4138072Z 
2022-11-23T02:20:42.4138141Z Generating XML reports...
2022-11-23T02:20:42.4138532Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021813.xml
2022-11-23T02:20:42.4138805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4138925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4139202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4139334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4139518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzty9brsy
2022-11-23T02:20:42.4139709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzty9brsy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4139949Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4139963Z 
2022-11-23T02:20:42.4140023Z Running tests...
2022-11-23T02:20:42.4140215Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4140480Z   test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29041
2022-11-23T02:20:42.4140700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29042
2022-11-23T02:20:42.4140891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29043
2022-11-23T02:20:42.4141046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29044
2022-11-23T02:20:42.4141318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4141441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4141707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4141839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4142060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxccdgs5b
2022-11-23T02:20:42.4142252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxccdgs5b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4142523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4142642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4142919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4143048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4143218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh37lyno
2022-11-23T02:20:42.4143406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh37lyno/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4143676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4143794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4144073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4144206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4144388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v6eq4op
2022-11-23T02:20:42.4144576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v6eq4op/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4144841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4144949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4145220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4145351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4145530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmpl4qt6
2022-11-23T02:20:42.4145722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmpl4qt6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4145882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4146040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4146265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4146413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4146475Z ok (2.574s)
2022-11-23T02:20:42.4146481Z 
2022-11-23T02:20:42.4146673Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4146745Z Ran 1 test in 2.575s
2022-11-23T02:20:42.4146750Z 
2022-11-23T02:20:42.4146805Z OK
2022-11-23T02:20:42.4146809Z 
2022-11-23T02:20:42.4146888Z Generating XML reports...
2022-11-23T02:20:42.4147308Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021817.xml
2022-11-23T02:20:42.4147577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4147697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4147965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4148097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4148278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5e8n2f0q
2022-11-23T02:20:42.4148500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5e8n2f0q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4148752Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4148757Z 
2022-11-23T02:20:42.4148826Z Running tests...
2022-11-23T02:20:42.4149021Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4149276Z   test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29300
2022-11-23T02:20:42.4149430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29301
2022-11-23T02:20:42.4149570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29302
2022-11-23T02:20:42.4149716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29303
2022-11-23T02:20:42.4149988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4150109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4150389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4150523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4150704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvouk4pw7
2022-11-23T02:20:42.4150895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvouk4pw7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4151156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4151272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4151547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4151681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4151862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpps_wr7aa
2022-11-23T02:20:42.4152055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpps_wr7aa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4152324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4152444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4152720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4152841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4153018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zowbied
2022-11-23T02:20:42.4153211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zowbied/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4153478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4153596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4153900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4154030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4154209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb43tt2ta
2022-11-23T02:20:42.4154499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb43tt2ta/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4154660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4154822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4155011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4155165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4155230Z ok (2.574s)
2022-11-23T02:20:42.4155235Z 
2022-11-23T02:20:42.4155432Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4155504Z Ran 1 test in 2.574s
2022-11-23T02:20:42.4155510Z 
2022-11-23T02:20:42.4155567Z OK
2022-11-23T02:20:42.4155571Z 
2022-11-23T02:20:42.4155641Z Generating XML reports...
2022-11-23T02:20:42.4156027Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021822.xml
2022-11-23T02:20:42.4156295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4156416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4156692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4156823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4157007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbhoa48o
2022-11-23T02:20:42.4157198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbhoa48o/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4157441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4157457Z 
2022-11-23T02:20:42.4157514Z Running tests...
2022-11-23T02:20:42.4157707Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4157956Z   test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29547
2022-11-23T02:20:42.4158267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29548
2022-11-23T02:20:42.4158416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29549
2022-11-23T02:20:42.4158567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29550
2022-11-23T02:20:42.4158843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4158964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4159233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4159364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4159546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0uftfnz
2022-11-23T02:20:42.4159737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0uftfnz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4160005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4160123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4160432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4160563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4160734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycj9tn57
2022-11-23T02:20:42.4160924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycj9tn57/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4161191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4161309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4161612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4161741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4161922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p5bm_ss
2022-11-23T02:20:42.4162114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p5bm_ss/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4162379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4162488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4162762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4162890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4163070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_dp6err
2022-11-23T02:20:42.4163260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_dp6err/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4163419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4163582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4163739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4163896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4163949Z ok (2.581s)
2022-11-23T02:20:42.4163954Z 
2022-11-23T02:20:42.4164145Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4164284Z Ran 1 test in 2.581s
2022-11-23T02:20:42.4164290Z 
2022-11-23T02:20:42.4164349Z OK
2022-11-23T02:20:42.4164355Z 
2022-11-23T02:20:42.4164435Z Generating XML reports...
2022-11-23T02:20:42.4164824Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021826.xml
2022-11-23T02:20:42.4165165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4165288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4165554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4165688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4165868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq0xen7v
2022-11-23T02:20:42.4166059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq0xen7v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4166309Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4166316Z 
2022-11-23T02:20:42.4166383Z Running tests...
2022-11-23T02:20:42.4166576Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4166870Z   test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29794
2022-11-23T02:20:42.4167026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29795
2022-11-23T02:20:42.4167234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29796
2022-11-23T02:20:42.4167384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29797
2022-11-23T02:20:42.4167659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4167780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4168089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4168220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4168405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfzicghk
2022-11-23T02:20:42.4168599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfzicghk/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4168857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4168975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4169252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4169382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4169565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbwhnlhs
2022-11-23T02:20:42.4169760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbwhnlhs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4170029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4170151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4170428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4170548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4170726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpgobvaz
2022-11-23T02:20:42.4170915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpgobvaz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4171180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4171302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4171575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4171707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4171888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tcy1wzh
2022-11-23T02:20:42.4172069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tcy1wzh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4172229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4172385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4172542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4172701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4172764Z ok (2.574s)
2022-11-23T02:20:42.4172769Z 
2022-11-23T02:20:42.4172958Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4173027Z Ran 1 test in 2.574s
2022-11-23T02:20:42.4173032Z 
2022-11-23T02:20:42.4173088Z OK
2022-11-23T02:20:42.4173093Z 
2022-11-23T02:20:42.4173190Z Generating XML reports...
2022-11-23T02:20:42.4173584Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021831.xml
2022-11-23T02:20:42.4173852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4173971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4174249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4174405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4174585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d9k55y7
2022-11-23T02:20:42.4174775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d9k55y7/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4175179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4175195Z 
2022-11-23T02:20:42.4175254Z Running tests...
2022-11-23T02:20:42.4175449Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4175709Z   test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30041
2022-11-23T02:20:42.4175861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30042
2022-11-23T02:20:42.4176131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30043
2022-11-23T02:20:42.4176282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30044
2022-11-23T02:20:42.4176554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4176673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4176943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4177074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4177254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmusm7gvb
2022-11-23T02:20:42.4177444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmusm7gvb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4177714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4177835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4178112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4178243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4178422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rxq_gmd
2022-11-23T02:20:42.4178604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rxq_gmd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4178870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4178987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4179261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4179393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4179574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1n1cth6
2022-11-23T02:20:42.4179762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1n1cth6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4180086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4180197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4180472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4180603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4180783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmson83y
2022-11-23T02:20:42.4180972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmson83y/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4181205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4181365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4181527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4181755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4181811Z ok (2.774s)
2022-11-23T02:20:42.4181816Z 
2022-11-23T02:20:42.4182010Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4182081Z Ran 1 test in 2.774s
2022-11-23T02:20:42.4182086Z 
2022-11-23T02:20:42.4182142Z OK
2022-11-23T02:20:42.4182146Z 
2022-11-23T02:20:42.4182226Z Generating XML reports...
2022-11-23T02:20:42.4182614Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021835.xml
2022-11-23T02:20:42.4182886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4183006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4183275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4183404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4183584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5shoiug
2022-11-23T02:20:42.4183774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5shoiug/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4184024Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4184030Z 
2022-11-23T02:20:42.4184097Z Running tests...
2022-11-23T02:20:42.4184290Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4184543Z   test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30459
2022-11-23T02:20:42.4184697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30460
2022-11-23T02:20:42.4184838Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30461
2022-11-23T02:20:42.4184989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30462
2022-11-23T02:20:42.4185259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4185376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4185653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4185782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4185964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjtbc1i5
2022-11-23T02:20:42.4186152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjtbc1i5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4186448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4186557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4186833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4186964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4187143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23vryj53
2022-11-23T02:20:42.4187333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23vryj53/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4187641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4187758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4188036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4188157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4188335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4__0hsc_
2022-11-23T02:20:42.4188518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4__0hsc_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4188785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4188903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4189177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4189313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4189491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldm_72bq
2022-11-23T02:20:42.4189681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldm_72bq/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4189831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4189990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4190147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4190303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4190366Z ok (2.476s)
2022-11-23T02:20:42.4190371Z 
2022-11-23T02:20:42.4190567Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4190638Z Ran 1 test in 2.477s
2022-11-23T02:20:42.4190643Z 
2022-11-23T02:20:42.4190700Z OK
2022-11-23T02:20:42.4190704Z 
2022-11-23T02:20:42.4190774Z Generating XML reports...
2022-11-23T02:20:42.4191163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021840.xml
2022-11-23T02:20:42.4191434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4191556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4191834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4191964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4192147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyagges_6
2022-11-23T02:20:42.4192343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyagges_6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4192594Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4192599Z 
2022-11-23T02:20:42.4192658Z Running tests...
2022-11-23T02:20:42.4192877Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4193140Z   test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30682
2022-11-23T02:20:42.4193291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30683
2022-11-23T02:20:42.4193547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30684
2022-11-23T02:20:42.4193699Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30685
2022-11-23T02:20:42.4194008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4194129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4194400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4194530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4194713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb5rh5nx
2022-11-23T02:20:42.4194901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb5rh5nx/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4195170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4195288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4195566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4195697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4195878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnokhu7y0
2022-11-23T02:20:42.4196061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnokhu7y0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4196328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4196448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4196723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4196853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4197031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz8y33bp
2022-11-23T02:20:42.4197223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz8y33bp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4197490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4197610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4197877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4198009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4198188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefxqduxz
2022-11-23T02:20:42.4198378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefxqduxz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4198535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4198695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4198855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4199012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4199065Z ok (2.674s)
2022-11-23T02:20:42.4199070Z 
2022-11-23T02:20:42.4199291Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4199363Z Ran 1 test in 2.674s
2022-11-23T02:20:42.4199368Z 
2022-11-23T02:20:42.4199423Z OK
2022-11-23T02:20:42.4199428Z 
2022-11-23T02:20:42.4199506Z Generating XML reports...
2022-11-23T02:20:42.4199890Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021844.xml
2022-11-23T02:20:42.4200163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4200310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4200590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4200712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4200897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2emih_e
2022-11-23T02:20:42.4201091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2emih_e/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4201342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4201347Z 
2022-11-23T02:20:42.4201413Z Running tests...
2022-11-23T02:20:42.4206799Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4207096Z   test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30917
2022-11-23T02:20:42.4207260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30918
2022-11-23T02:20:42.4207413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30919
2022-11-23T02:20:42.4207564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30920
2022-11-23T02:20:42.4207863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4207975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4208256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4208388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4208571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2g00q2vn
2022-11-23T02:20:42.4208768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2g00q2vn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4209038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4209158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4209436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4209571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4209741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp453j9_dy
2022-11-23T02:20:42.4209933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp453j9_dy/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4210201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4210326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4210605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4210735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4210917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0o8xwh1
2022-11-23T02:20:42.4211261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0o8xwh1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4211525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4211647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4211922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4212054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4212275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpundrtwlf
2022-11-23T02:20:42.4212468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpundrtwlf/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4212632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4212796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4212956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4213103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4213168Z ok (2.576s)
2022-11-23T02:20:42.4213175Z 
2022-11-23T02:20:42.4213374Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4213446Z Ran 1 test in 2.576s
2022-11-23T02:20:42.4213451Z 
2022-11-23T02:20:42.4213511Z OK
2022-11-23T02:20:42.4213516Z 
2022-11-23T02:20:42.4213593Z Generating XML reports...
2022-11-23T02:20:42.4213990Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021849.xml
2022-11-23T02:20:42.4214265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4214377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4214658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4214794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4215076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwiuxn4aa
2022-11-23T02:20:42.4215273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwiuxn4aa/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4215528Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4215533Z 
2022-11-23T02:20:42.4215604Z Running tests...
2022-11-23T02:20:42.4215802Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4216070Z   test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31152
2022-11-23T02:20:42.4216213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31153
2022-11-23T02:20:42.4216365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31154
2022-11-23T02:20:42.4216516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31155
2022-11-23T02:20:42.4216791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4216911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4217197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4217332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4217519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6lixbs5
2022-11-23T02:20:42.4217765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6lixbs5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4218027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4218147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4218521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4218653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4218877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75_y4rkw
2022-11-23T02:20:42.4219067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75_y4rkw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4219338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4219461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4219727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4219858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4220040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbue929b
2022-11-23T02:20:42.4220229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbue929b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4220505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4220626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4220903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4221035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4221217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgr0839r
2022-11-23T02:20:42.4221396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgr0839r/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4221559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4221721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4221880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4222041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4222106Z ok (2.576s)
2022-11-23T02:20:42.4222111Z 
2022-11-23T02:20:42.4222308Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4222384Z Ran 1 test in 2.577s
2022-11-23T02:20:42.4222389Z 
2022-11-23T02:20:42.4222438Z OK
2022-11-23T02:20:42.4222457Z 
2022-11-23T02:20:42.4222594Z Generating XML reports...
2022-11-23T02:20:42.4222985Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021853.xml
2022-11-23T02:20:42.4223254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4223374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4223649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4223784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4223966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0y3ni2dl
2022-11-23T02:20:42.4224156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0y3ni2dl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4224431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4224436Z 
2022-11-23T02:20:42.4224507Z Running tests...
2022-11-23T02:20:42.4224702Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4224969Z   test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31387
2022-11-23T02:20:42.4225125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31388
2022-11-23T02:20:42.4225308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31389
2022-11-23T02:20:42.4225457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31390
2022-11-23T02:20:42.4225728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4225851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4226116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4226246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4226428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yp2ozbb
2022-11-23T02:20:42.4226621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yp2ozbb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4226890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4227017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4227292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4227427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4227602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrpyyy4t
2022-11-23T02:20:42.4227795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrpyyy4t/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4228065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4228184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4228461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4228596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4228775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rqcbs3c
2022-11-23T02:20:42.4228967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rqcbs3c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4229236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4229347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4229621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4229829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4230011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d_4i5zu
2022-11-23T02:20:42.4230200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d_4i5zu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4230364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4230523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4230712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4230862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4230929Z ok (2.575s)
2022-11-23T02:20:42.4230934Z 
2022-11-23T02:20:42.4231133Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4231206Z Ran 1 test in 2.575s
2022-11-23T02:20:42.4231211Z 
2022-11-23T02:20:42.4231267Z OK
2022-11-23T02:20:42.4231272Z 
2022-11-23T02:20:42.4231352Z Generating XML reports...
2022-11-23T02:20:42.4231740Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021858.xml
2022-11-23T02:20:42.4232045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4232164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4232431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4232562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4232743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4zf2xah
2022-11-23T02:20:42.4232933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4zf2xah/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4233185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4233190Z 
2022-11-23T02:20:42.4233259Z Running tests...
2022-11-23T02:20:42.4233457Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4233720Z   test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31622
2022-11-23T02:20:42.4233873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31623
2022-11-23T02:20:42.4234016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31624
2022-11-23T02:20:42.4234164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31625
2022-11-23T02:20:42.4234432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4234552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4234828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4234959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4235141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd4iammn
2022-11-23T02:20:42.4235335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd4iammn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4235595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4235835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4236115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4236245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4236428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3i3stma
2022-11-23T02:20:42.4236617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3i3stma/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4236890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4237009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4237319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4237441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4237620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpj_oa0q
2022-11-23T02:20:42.4237811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpj_oa0q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4238080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4238199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4238516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4238648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4238828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ebfkbyz
2022-11-23T02:20:42.4239011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ebfkbyz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4239173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4239331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4239487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4239648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4239710Z ok (4.578s)
2022-11-23T02:20:42.4239717Z 
2022-11-23T02:20:42.4239913Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4239985Z Ran 1 test in 4.578s
2022-11-23T02:20:42.4239991Z 
2022-11-23T02:20:42.4240048Z OK
2022-11-23T02:20:42.4240053Z 
2022-11-23T02:20:42.4240122Z Generating XML reports...
2022-11-23T02:20:42.4240514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021902.xml
2022-11-23T02:20:42.4240853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4240973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4241247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4241377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4241561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotl0dqu1
2022-11-23T02:20:42.4241754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotl0dqu1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4241994Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4242010Z 
2022-11-23T02:20:42.4242070Z Running tests...
2022-11-23T02:20:42.4242266Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4242529Z   test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31845
2022-11-23T02:20:42.4242683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31846
2022-11-23T02:20:42.4242832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31847
2022-11-23T02:20:42.4242983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31848
2022-11-23T02:20:42.4243258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4243375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4243642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4243800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4243983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvtkttoh
2022-11-23T02:20:42.4244174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvtkttoh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4244443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4244562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4244865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4245073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4245247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu70nyzym
2022-11-23T02:20:42.4245440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu70nyzym/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4245712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4245832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4246108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4246237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4246414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qogcj15
2022-11-23T02:20:42.4246605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qogcj15/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4246872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4246982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4247264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4247393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4247573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy72ywhap
2022-11-23T02:20:42.4247765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy72ywhap/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4247925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4248087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4248247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4248411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4248466Z ok (4.577s)
2022-11-23T02:20:42.4248473Z 
2022-11-23T02:20:42.4248664Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4248735Z Ran 1 test in 4.578s
2022-11-23T02:20:42.4248741Z 
2022-11-23T02:20:42.4248794Z OK
2022-11-23T02:20:42.4248798Z 
2022-11-23T02:20:42.4248881Z Generating XML reports...
2022-11-23T02:20:42.4249271Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021909.xml
2022-11-23T02:20:42.4249545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4249740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4250009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4250141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4250357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0vqez5n
2022-11-23T02:20:42.4250551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0vqez5n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4250805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4250810Z 
2022-11-23T02:20:42.4250880Z Running tests...
2022-11-23T02:20:42.4251075Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4251325Z   test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32068
2022-11-23T02:20:42.4251515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32069
2022-11-23T02:20:42.4251657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32070
2022-11-23T02:20:42.4251812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32071
2022-11-23T02:20:42.4252085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4252204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4252478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4252608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4252795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgk3ybi5
2022-11-23T02:20:42.4252993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgk3ybi5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4253254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4253373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4253651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4253782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4253960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87n8zuli
2022-11-23T02:20:42.4254148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87n8zuli/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4254414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4254533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4254812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4255079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4255280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_44byew
2022-11-23T02:20:42.4255472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_44byew/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4255742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4255862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4256135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4256269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4256449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82erxssr
2022-11-23T02:20:42.4256628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82erxssr/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4256844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4257006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4257166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4257325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4257596Z [W tensorpipe_agent.cpp:939] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4257900Z [W tensorpipe_agent.cpp:939] RPC agent for worker2 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4258192Z [W tensorpipe_agent.cpp:624] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down
2022-11-23T02:20:42.4258476Z [W tensorpipe_agent.cpp:624] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down
2022-11-23T02:20:42.4258755Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down
2022-11-23T02:20:42.4259024Z [W tensorpipe_agent.cpp:624] RPC agent for worker3 won't send response to request #30 to worker2, as the agent is shutting down
2022-11-23T02:20:42.4259090Z ok (15.593s)
2022-11-23T02:20:42.4259096Z 
2022-11-23T02:20:42.4259291Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4259365Z Ran 1 test in 15.593s
2022-11-23T02:20:42.4259370Z 
2022-11-23T02:20:42.4259426Z OK
2022-11-23T02:20:42.4259431Z 
2022-11-23T02:20:42.4259513Z Generating XML reports...
2022-11-23T02:20:42.4259996Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021915.xml
2022-11-23T02:20:42.4260273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4260397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4260668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4260797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4260979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ncrnhi2
2022-11-23T02:20:42.4261170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ncrnhi2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4261423Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4261428Z 
2022-11-23T02:20:42.4261498Z Running tests...
2022-11-23T02:20:42.4261691Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4261952Z   test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32291
2022-11-23T02:20:42.4262104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32292
2022-11-23T02:20:42.4262247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32293
2022-11-23T02:20:42.4262397Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32294
2022-11-23T02:20:42.4262666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4262788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4263063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4263196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4263413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe208k466
2022-11-23T02:20:42.4263608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe208k466/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4263867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4263985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4264260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4264420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4264599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugc79dgi
2022-11-23T02:20:42.4264789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugc79dgi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4265062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4265182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4265461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4265581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4265761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24elpuzb
2022-11-23T02:20:42.4265951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24elpuzb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4266293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4266414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4266696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4266830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4267013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_96q4dsh
2022-11-23T02:20:42.4267191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_96q4dsh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4267351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4267511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4267673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4267828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4267894Z ok (2.574s)
2022-11-23T02:20:42.4267899Z 
2022-11-23T02:20:42.4268093Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4268167Z Ran 1 test in 2.574s
2022-11-23T02:20:42.4268172Z 
2022-11-23T02:20:42.4268230Z OK
2022-11-23T02:20:42.4268234Z 
2022-11-23T02:20:42.4268303Z Generating XML reports...
2022-11-23T02:20:42.4268689Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021932.xml
2022-11-23T02:20:42.4268960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4269080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4269361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4269494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4269676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0skkbzg1
2022-11-23T02:20:42.4269898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0skkbzg1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4270139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4270153Z 
2022-11-23T02:20:42.4270211Z Running tests...
2022-11-23T02:20:42.4270404Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4270673Z   test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32526
2022-11-23T02:20:42.4270852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32527
2022-11-23T02:20:42.4271007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32528
2022-11-23T02:20:42.4271156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32529
2022-11-23T02:20:42.4271430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4271551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4271819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4271950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4272131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeapfbwro
2022-11-23T02:20:42.4272323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeapfbwro/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4272592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4272713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4272990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4273124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4273303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl7rdjxw
2022-11-23T02:20:42.4273484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl7rdjxw/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4273751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4273869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4274147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4274276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4274455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20_nr7mi
2022-11-23T02:20:42.4274643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20_nr7mi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4274910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4275018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4275290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4275420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4275601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvpr8tl_
2022-11-23T02:20:42.4275791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvpr8tl_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4275950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4276143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4276303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4276462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4276516Z ok (2.576s)
2022-11-23T02:20:42.4276521Z 
2022-11-23T02:20:42.4276713Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4276784Z Ran 1 test in 2.576s
2022-11-23T02:20:42.4276788Z 
2022-11-23T02:20:42.4276845Z OK
2022-11-23T02:20:42.4276850Z 
2022-11-23T02:20:42.4276929Z Generating XML reports...
2022-11-23T02:20:42.4277349Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021937.xml
2022-11-23T02:20:42.4277622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4277746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4278016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4278150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4278331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3d1niz2
2022-11-23T02:20:42.4278523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3d1niz2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4278773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4278781Z 
2022-11-23T02:20:42.4278852Z Running tests...
2022-11-23T02:20:42.4279048Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4279313Z   test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32761
2022-11-23T02:20:42.4279469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32762
2022-11-23T02:20:42.4279695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32763
2022-11-23T02:20:42.4279848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32764
2022-11-23T02:20:42.4280120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4280238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4280513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4280647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4280829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzieyou_x
2022-11-23T02:20:42.4281024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzieyou_x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4281292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4281401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4281677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4281809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4281990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiz8csgj4
2022-11-23T02:20:42.4282182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiz8csgj4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4282451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4282572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4282877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4282998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4283179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hy627o1
2022-11-23T02:20:42.4283364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hy627o1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4283634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4283796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4284073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4284206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4284387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t17hvbn
2022-11-23T02:20:42.4284577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t17hvbn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4284727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4284887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4285194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4285355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4285423Z ok (2.576s)
2022-11-23T02:20:42.4285429Z 
2022-11-23T02:20:42.4285628Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4285700Z Ran 1 test in 2.576s
2022-11-23T02:20:42.4285706Z 
2022-11-23T02:20:42.4285765Z OK
2022-11-23T02:20:42.4285769Z 
2022-11-23T02:20:42.4285842Z Generating XML reports...
2022-11-23T02:20:42.4286229Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021941.xml
2022-11-23T02:20:42.4286500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4286619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4286893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4287028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4287212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_exwow8
2022-11-23T02:20:42.4287405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_exwow8/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4287658Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4287664Z 
2022-11-23T02:20:42.4287722Z Running tests...
2022-11-23T02:20:42.4287916Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4288187Z   test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 528
2022-11-23T02:20:42.4288337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 529
2022-11-23T02:20:42.4288484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 530
2022-11-23T02:20:42.4288634Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 531
2022-11-23T02:20:42.4288904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4289025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4289367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4289499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4289680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32qxo_jb
2022-11-23T02:20:42.4289866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32qxo_jb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4290133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4290281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4290559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4290693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4290874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0awwyld
2022-11-23T02:20:42.4291055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0awwyld/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4291321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4291437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4291714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4291843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4292024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dj2a5g4
2022-11-23T02:20:42.4292210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dj2a5g4/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4292479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4292598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4292865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4292995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4293176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5nu9qh2
2022-11-23T02:20:42.4293366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5nu9qh2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4293529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4293689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4293846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4294078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4294132Z ok (2.574s)
2022-11-23T02:20:42.4294137Z 
2022-11-23T02:20:42.4294332Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4294403Z Ran 1 test in 2.575s
2022-11-23T02:20:42.4294408Z 
2022-11-23T02:20:42.4294465Z OK
2022-11-23T02:20:42.4294469Z 
2022-11-23T02:20:42.4294548Z Generating XML reports...
2022-11-23T02:20:42.4295024Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021946.xml
2022-11-23T02:20:42.4295301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4295422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4295704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4295875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4296060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr5j3j3b
2022-11-23T02:20:42.4296253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr5j3j3b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4296505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4296511Z 
2022-11-23T02:20:42.4296580Z Running tests...
2022-11-23T02:20:42.4296775Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4297083Z   test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 765
2022-11-23T02:20:42.4297235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 766
2022-11-23T02:20:42.4297376Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 767
2022-11-23T02:20:42.4297526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 768
2022-11-23T02:20:42.4297877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4298000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4298279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4298414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4298601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmf87q49
2022-11-23T02:20:42.4298794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmf87q49/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4299065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4299173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4299450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4299581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4299762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnwl2epn
2022-11-23T02:20:42.4299953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnwl2epn/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4300224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4300409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4300683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4300817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4300988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas7qo0iu
2022-11-23T02:20:42.4301177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas7qo0iu/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4301445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4301561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4301832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4301965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4302146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyuyiujc
2022-11-23T02:20:42.4302369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyuyiujc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4302518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4302678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4302832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4302988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4303052Z ok (2.674s)
2022-11-23T02:20:42.4303058Z 
2022-11-23T02:20:42.4303281Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4303354Z Ran 1 test in 2.674s
2022-11-23T02:20:42.4303359Z 
2022-11-23T02:20:42.4303417Z OK
2022-11-23T02:20:42.4303422Z 
2022-11-23T02:20:42.4303491Z Generating XML reports...
2022-11-23T02:20:42.4303884Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021950.xml
2022-11-23T02:20:42.4304152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4304272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4304552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4304683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4304868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexm8kpnh
2022-11-23T02:20:42.4305061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexm8kpnh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4305312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4305317Z 
2022-11-23T02:20:42.4305375Z Running tests...
2022-11-23T02:20:42.4305569Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4305820Z   test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1000
2022-11-23T02:20:42.4305972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1001
2022-11-23T02:20:42.4306119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1002
2022-11-23T02:20:42.4306268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1003
2022-11-23T02:20:42.4306538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4306659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4306937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4307060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4307241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprepmuza
2022-11-23T02:20:42.4307432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprepmuza/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4307765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4307884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4308159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4308293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4308473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbks20l53
2022-11-23T02:20:42.4308652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbks20l53/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4308946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4309064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4309339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4309469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4309649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyamm94c
2022-11-23T02:20:42.4309874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyamm94c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4310142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4310259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4310527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4310657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4310836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k79ham5
2022-11-23T02:20:42.4311028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k79ham5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4311191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4311352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4311512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4311668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4311721Z ok (2.777s)
2022-11-23T02:20:42.4311734Z 
2022-11-23T02:20:42.4311917Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4311990Z Ran 1 test in 2.777s
2022-11-23T02:20:42.4311995Z 
2022-11-23T02:20:42.4312054Z OK
2022-11-23T02:20:42.4312059Z 
2022-11-23T02:20:42.4312140Z Generating XML reports...
2022-11-23T02:20:42.4312526Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021955.xml
2022-11-23T02:20:42.4312798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4312922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4313201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4313323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4313508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez0elep_
2022-11-23T02:20:42.4313704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez0elep_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4313951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4313956Z 
2022-11-23T02:20:42.4314025Z Running tests...
2022-11-23T02:20:42.4314216Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4314478Z   test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1235
2022-11-23T02:20:42.4314635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1236
2022-11-23T02:20:42.4314783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1237
2022-11-23T02:20:42.4314920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1238
2022-11-23T02:20:42.4315229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4315350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4315631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4315763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4315944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yqufd61
2022-11-23T02:20:42.4316138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yqufd61/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4316439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4316550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4316826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4316955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4317136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvipv2kz
2022-11-23T02:20:42.4317329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvipv2kz/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4317597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4317717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4317992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4318126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4318298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjtf22i0
2022-11-23T02:20:42.4318489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjtf22i0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4318843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4318960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4319236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4319369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4319552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6t1jl4c
2022-11-23T02:20:42.4319742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6t1jl4c/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4319892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4320053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4320211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4320366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4320430Z ok (2.474s)
2022-11-23T02:20:42.4320435Z 
2022-11-23T02:20:42.4320633Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4320706Z Ran 1 test in 2.475s
2022-11-23T02:20:42.4320711Z 
2022-11-23T02:20:42.4320769Z OK
2022-11-23T02:20:42.4320776Z 
2022-11-23T02:20:42.4320855Z Generating XML reports...
2022-11-23T02:20:42.4321234Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021959.xml
2022-11-23T02:20:42.4321502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4321653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4321929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4322058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4322242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcrrellc
2022-11-23T02:20:42.4322432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcrrellc/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4322712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4322717Z 
2022-11-23T02:20:42.4322787Z Running tests...
2022-11-23T02:20:42.4322971Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4323230Z   test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1502
2022-11-23T02:20:42.4323380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1503
2022-11-23T02:20:42.4323532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1504
2022-11-23T02:20:42.4323748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1505
2022-11-23T02:20:42.4324025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4324147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4324428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4324550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4324734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoat2x2zb
2022-11-23T02:20:42.4325001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoat2x2zb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4325279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4325398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4325679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4325812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4325993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99l8wh29
2022-11-23T02:20:42.4326187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99l8wh29/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4326442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4326564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4326839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4326971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4327151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bluk9bd
2022-11-23T02:20:42.4327344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bluk9bd/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4327612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4327734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4328000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4328131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4328348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd8x143u
2022-11-23T02:20:42.4328540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd8x143u/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4328698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4328857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4329013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4329214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4329328Z Rank 2 finished testing 20 times in 0.12411093711853027 seconds.
2022-11-23T02:20:42.4329430Z Rank 0 finished testing 20 times in 0.13238763809204102 seconds.
2022-11-23T02:20:42.4329541Z Rank 1 finished testing 20 times in 0.10614275932312012 seconds.
2022-11-23T02:20:42.4329653Z Rank 3 finished testing 20 times in 0.11314105987548828 seconds.
2022-11-23T02:20:42.4329716Z ok (2.574s)
2022-11-23T02:20:42.4329721Z 
2022-11-23T02:20:42.4329915Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4329987Z Ran 1 test in 2.575s
2022-11-23T02:20:42.4329991Z 
2022-11-23T02:20:42.4330047Z OK
2022-11-23T02:20:42.4330052Z 
2022-11-23T02:20:42.4330132Z Generating XML reports...
2022-11-23T02:20:42.4330514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022004.xml
2022-11-23T02:20:42.4330784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4330903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4331177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4331313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4331495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozoetq0v
2022-11-23T02:20:42.4331691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozoetq0v/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4331943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4331948Z 
2022-11-23T02:20:42.4332017Z Running tests...
2022-11-23T02:20:42.4332200Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4332483Z   test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1737
2022-11-23T02:20:42.4332635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1738
2022-11-23T02:20:42.4332785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1739
2022-11-23T02:20:42.4332932Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1740
2022-11-23T02:20:42.4333201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4333324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4333604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4333726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4333912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj27qure2
2022-11-23T02:20:42.4334105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj27qure2/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4334400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4334520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4334797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4335076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4335275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44_ok43q
2022-11-23T02:20:42.4335467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44_ok43q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4335889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4336011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4336285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4336419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4336598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2oy_5zh
2022-11-23T02:20:42.4336787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2oy_5zh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4337055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4337176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4337440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4337575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4337758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk426y7o5
2022-11-23T02:20:42.4337948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk426y7o5/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4338107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4338265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4338426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4338583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4338649Z ok (2.272s)
2022-11-23T02:20:42.4338656Z 
2022-11-23T02:20:42.4338841Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4338913Z Ran 1 test in 2.272s
2022-11-23T02:20:42.4338918Z 
2022-11-23T02:20:42.4338972Z OK
2022-11-23T02:20:42.4338976Z 
2022-11-23T02:20:42.4339057Z Generating XML reports...
2022-11-23T02:20:42.4339449Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022008.xml
2022-11-23T02:20:42.4339722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4339843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4340119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4340240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4340422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvntk3e9b
2022-11-23T02:20:42.4340618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvntk3e9b/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4340935Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4340941Z 
2022-11-23T02:20:42.4341009Z Running tests...
2022-11-23T02:20:42.4341245Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4341513Z   test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1796
2022-11-23T02:20:42.4341667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1797
2022-11-23T02:20:42.4341819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1798
2022-11-23T02:20:42.4341955Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1799
2022-11-23T02:20:42.4342258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4342380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4342659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4342792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4342976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22cu6d7x
2022-11-23T02:20:42.4343166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22cu6d7x/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4343433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4343553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4343819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4343953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4344135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ht9igcs
2022-11-23T02:20:42.4344325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ht9igcs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4344592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4344709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4344987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4345119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4345289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyhysuuh
2022-11-23T02:20:42.4345485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyhysuuh/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4345754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4345872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4346151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4346280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4346456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tr4iasm
2022-11-23T02:20:42.4346643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tr4iasm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4346802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4346953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4347110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4347266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4347330Z ok (2.478s)
2022-11-23T02:20:42.4347335Z 
2022-11-23T02:20:42.4347551Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4347625Z Ran 1 test in 2.478s
2022-11-23T02:20:42.4347630Z 
2022-11-23T02:20:42.4347686Z OK
2022-11-23T02:20:42.4347691Z 
2022-11-23T02:20:42.4347770Z Generating XML reports...
2022-11-23T02:20:42.4348147Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022012.xml
2022-11-23T02:20:42.4348420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4348570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4348846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4348977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4349158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6aac77q
2022-11-23T02:20:42.4349346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6aac77q/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4349596Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4349601Z 
2022-11-23T02:20:42.4349670Z Running tests...
2022-11-23T02:20:42.4349853Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4350106Z   test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2019
2022-11-23T02:20:42.4350257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2020
2022-11-23T02:20:42.4350403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2021
2022-11-23T02:20:42.4350550Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2022
2022-11-23T02:20:42.4350822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4350943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4351220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4351341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4351527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppshtqnih
2022-11-23T02:20:42.4351721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppshtqnih/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4351990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4352111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4352387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4352518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4352696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1uqnez6
2022-11-23T02:20:42.4352887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1uqnez6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4353225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4353346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4353624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4353756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4353936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu0c7gfi
2022-11-23T02:20:42.4354157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu0c7gfi/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4354426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4354544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4354820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4354941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4355149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmid2adm
2022-11-23T02:20:42.4355336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmid2adm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4355494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4355654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4355814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4355970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4356145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4356307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4356480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.4356646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.4357010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4357312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4357601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4357887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4358150Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4358412Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4358679Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4358940Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4359185Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4359434Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4359501Z ok (2.777s)
2022-11-23T02:20:42.4359506Z 
2022-11-23T02:20:42.4359700Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4359770Z Ran 1 test in 2.777s
2022-11-23T02:20:42.4359775Z 
2022-11-23T02:20:42.4359834Z OK
2022-11-23T02:20:42.4359839Z 
2022-11-23T02:20:42.4359955Z Generating XML reports...
2022-11-23T02:20:42.4360345Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022017.xml
2022-11-23T02:20:42.4360620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4360730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4361009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4361166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4361349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjmfybjt
2022-11-23T02:20:42.4361541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjmfybjt/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4361798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4361803Z 
2022-11-23T02:20:42.4361873Z Running tests...
2022-11-23T02:20:42.4362064Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4362323Z   test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2266
2022-11-23T02:20:42.4362466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2267
2022-11-23T02:20:42.4362615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2268
2022-11-23T02:20:42.4362765Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2269
2022-11-23T02:20:42.4363036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4363156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4363437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4363568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4363748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp193wdfy1
2022-11-23T02:20:42.4363999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp193wdfy1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4364273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4364393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4364669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4364797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4365040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc70lurfl
2022-11-23T02:20:42.4365235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc70lurfl/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4365502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4365622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4365888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4366021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4366201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcyci3sj
2022-11-23T02:20:42.4366389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcyci3sj/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4366688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4366809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4367085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4367216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4367394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_v8oh27
2022-11-23T02:20:42.4367570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_v8oh27/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4367829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4367989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4368146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4368306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4368482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4368655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4368826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:20:42.4368986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:20:42.4369289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4369581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4369872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4370160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:20:42.4370426Z [W tensorpipe_agent.cpp:725] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4370691Z [W tensorpipe_agent.cpp:725] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4371014Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4371270Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4371515Z [W tensorpipe_agent.cpp:725] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:42.4371767Z [W tensorpipe_agent.cpp:725] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-11-23T02:20:42.4371829Z ok (2.777s)
2022-11-23T02:20:42.4371835Z 
2022-11-23T02:20:42.4372022Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4372094Z Ran 1 test in 2.777s
2022-11-23T02:20:42.4372100Z 
2022-11-23T02:20:42.4372156Z OK
2022-11-23T02:20:42.4372161Z 
2022-11-23T02:20:42.4372242Z Generating XML reports...
2022-11-23T02:20:42.4372631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022021.xml
2022-11-23T02:20:42.4372939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4373061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4373339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4373461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4373647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxtsy872
2022-11-23T02:20:42.4373885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxtsy872/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4374135Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4374140Z 
2022-11-23T02:20:42.4374209Z Running tests...
2022-11-23T02:20:42.4374405Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4374657Z   test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2513
2022-11-23T02:20:42.4374808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2514
2022-11-23T02:20:42.4375054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2515
2022-11-23T02:20:42.4375195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2516
2022-11-23T02:20:42.4375469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4375592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4375870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4376002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4376190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1fef9c_
2022-11-23T02:20:42.4376389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1fef9c_/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4376660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4376769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4377047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4377181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4377364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu1xmmau
2022-11-23T02:20:42.4377555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu1xmmau/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4377898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4378021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4378296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4378428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4378598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaz6ajjo
2022-11-23T02:20:42.4378792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaz6ajjo/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4379064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4379184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4379517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4379653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4379835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k1isg36
2022-11-23T02:20:42.4380026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k1isg36/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4380188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:42.4380339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4380534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4380692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4380756Z ok (2.273s)
2022-11-23T02:20:42.4380762Z 
2022-11-23T02:20:42.4380962Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4381035Z Ran 1 test in 2.273s
2022-11-23T02:20:42.4381040Z 
2022-11-23T02:20:42.4381099Z OK
2022-11-23T02:20:42.4381104Z 
2022-11-23T02:20:42.4381183Z Generating XML reports...
2022-11-23T02:20:42.4381561Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022026.xml
2022-11-23T02:20:42.4381832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4381952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4382231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4382362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4382547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepei_cs0
2022-11-23T02:20:42.4382739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepei_cs0/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4382992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4382997Z 
2022-11-23T02:20:42.4383066Z Running tests...
2022-11-23T02:20:42.4383251Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4383534Z   test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2622
2022-11-23T02:20:42.4383687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2623
2022-11-23T02:20:42.4383837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2624
2022-11-23T02:20:42.4384111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4384235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4384512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4384645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4384819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq65f_2n
2022-11-23T02:20:42.4385011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq65f_2n/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4385279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4385400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4385674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4385907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4386091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86kq49za
2022-11-23T02:20:42.4386282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86kq49za/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4386555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4386665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4386942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4387138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4387322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv628b0fs
2022-11-23T02:20:42.4387513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv628b0fs/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4387676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4387838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4387998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4388051Z ok (2.370s)
2022-11-23T02:20:42.4388068Z 
2022-11-23T02:20:42.4388254Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4388327Z Ran 1 test in 2.370s
2022-11-23T02:20:42.4388334Z 
2022-11-23T02:20:42.4388389Z OK
2022-11-23T02:20:42.4388394Z 
2022-11-23T02:20:42.4388476Z Generating XML reports...
2022-11-23T02:20:42.4388890Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022030.xml
2022-11-23T02:20:42.4389163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4389282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4389562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4389682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4389867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpxriqy1
2022-11-23T02:20:42.4390058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpxriqy1/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4390314Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4390319Z 
2022-11-23T02:20:42.4390389Z Running tests...
2022-11-23T02:20:42.4390650Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4390929Z   test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2792
2022-11-23T02:20:42.4391081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2793
2022-11-23T02:20:42.4391231Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2794
2022-11-23T02:20:42.4391494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4391615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4391892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4392022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4392205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct5wy4ap
2022-11-23T02:20:42.4392429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct5wy4ap/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4392701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4392820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4393085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4393217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4393394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22ar71n6
2022-11-23T02:20:42.4393610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22ar71n6/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4393879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4393999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4394278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4394409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4394591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh04uvhp
2022-11-23T02:20:42.4394772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh04uvhp/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4394930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4395091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4395247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4395312Z ok (2.371s)
2022-11-23T02:20:42.4395317Z 
2022-11-23T02:20:42.4395511Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4395586Z Ran 1 test in 2.371s
2022-11-23T02:20:42.4395591Z 
2022-11-23T02:20:42.4395646Z OK
2022-11-23T02:20:42.4395651Z 
2022-11-23T02:20:42.4395721Z Generating XML reports...
2022-11-23T02:20:42.4396197Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022034.xml
2022-11-23T02:20:42.4396471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4396589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4396867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4396998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4397180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpushgd4mm
2022-11-23T02:20:42.4397375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpushgd4mm/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4397625Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-11-23T02:20:42.4397630Z 
2022-11-23T02:20:42.4397689Z Running tests...
2022-11-23T02:20:42.4397880Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4398180Z   test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2962
2022-11-23T02:20:42.4398333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2963
2022-11-23T02:20:42.4398482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2964
2022-11-23T02:20:42.4398751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4398904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4399181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4399314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4399485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqg0u_ne
2022-11-23T02:20:42.4399678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqg0u_ne/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4399946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4400091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4400369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4400501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4400685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1pbvzke
2022-11-23T02:20:42.4400878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1pbvzke/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4401135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4401254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4401527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4401659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4401838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeeoemhjb
2022-11-23T02:20:42.4402030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeoemhjb/_remote_module_non_scriptable.py
2022-11-23T02:20:42.4402194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4402354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:42.4402510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4402769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqg0u_ne/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.4403041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeoemhjb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.4403318Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpeeoemhjb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:20:42.4403381Z ok (2.370s)
2022-11-23T02:20:42.4403387Z 
2022-11-23T02:20:42.4403581Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4403654Z Ran 1 test in 2.371s
2022-11-23T02:20:42.4403659Z 
2022-11-23T02:20:42.4403717Z OK
2022-11-23T02:20:42.4403721Z 
2022-11-23T02:20:42.4403801Z Generating XML reports...
2022-11-23T02:20:42.4404216Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022038.xml
2022-11-23T02:20:42.4404221Z 
2022-11-23T02:20:42.4404565Z ##[endgroup]
2022-11-23T02:20:42.4404999Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_x2yvtp4u)
2022-11-23T02:20:42.4405008Z 
2022-11-23T02:20:42.4405050Z 
2022-11-23T02:20:42.4405192Z real	76m44.998s
2022-11-23T02:20:42.4405259Z user	145m34.437s
2022-11-23T02:20:42.4405326Z sys	34m57.501s
2022-11-23T02:20:42.4405401Z + assert_git_not_dirty
2022-11-23T02:20:42.4405580Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2022-11-23T02:20:42.4405717Z + [[ linux-focal-py3.7-gcc7 != *xla* ]]
2022-11-23T02:20:42.4405816Z ++ git status --porcelain
2022-11-23T02:20:46.4439887Z + git_status=
2022-11-23T02:20:46.4440412Z + [[ -n '' ]]
2022-11-23T02:20:46.4440780Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-11-23T02:20:46.4441000Z + [[ 1 == 1 ]]
2022-11-23T02:20:46.4441151Z + test_rpc
2022-11-23T02:20:46.4441393Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2022-11-23T02:20:46.4441646Z + echo 'Testing RPC C++ tests'
2022-11-23T02:20:46.4441830Z Testing RPC C++ tests
2022-11-23T02:20:46.4442810Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-11-23T02:20:46.4485554Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-11-23T02:20:46.4493608Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin
2022-11-23T02:20:46.4501111Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-11-23T02:20:46.4501477Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-11-23T02:20:46.4546415Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-11-23T02:20:46.7012871Z CUDA not available. Disabling CUDA and MultiCUDA tests
2022-11-23T02:20:46.7013599Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA
2022-11-23T02:20:46.7013977Z [==========] Running 8 tests from 3 test suites.
2022-11-23T02:20:46.7014275Z [----------] Global test environment set-up.
2022-11-23T02:20:46.7014570Z [----------] 4 tests from WireSerialize
2022-11-23T02:20:46.7014981Z [ RUN      ] WireSerialize.Base
2022-11-23T02:20:46.7182537Z [       OK ] WireSerialize.Base (16 ms)
2022-11-23T02:20:46.7183025Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-11-23T02:20:46.7260529Z [       OK ] WireSerialize.RecopySparseTensors (7 ms)
2022-11-23T02:20:46.7261019Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-11-23T02:20:46.7333373Z [       OK ] WireSerialize.CloneSparseTensors (7 ms)
2022-11-23T02:20:46.7333940Z [ RUN      ] WireSerialize.Errors
2022-11-23T02:20:46.7357269Z [       OK ] WireSerialize.Errors (2 ms)
2022-11-23T02:20:46.7357860Z [----------] 4 tests from WireSerialize (34 ms total)
2022-11-23T02:20:46.7358161Z 
2022-11-23T02:20:46.7358362Z [----------] 1 test from TestE2ETensorPipe
2022-11-23T02:20:46.7358672Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-11-23T02:20:47.1135606Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:20:47.1145601Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (378 ms)
2022-11-23T02:20:47.1146322Z [----------] 1 test from TestE2ETensorPipe (378 ms total)
2022-11-23T02:20:47.1146504Z 
2022-11-23T02:20:47.1146723Z [----------] 3 tests from TensorpipeSerialize
2022-11-23T02:20:47.1147304Z [ RUN      ] TensorpipeSerialize.Base
2022-11-23T02:20:47.1147660Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-11-23T02:20:47.1148011Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-11-23T02:20:47.1212928Z [       OK ] TensorpipeSerialize.RecopySparseTensors (6 ms)
2022-11-23T02:20:47.1213508Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-11-23T02:20:47.1213993Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-11-23T02:20:47.1214666Z [----------] 3 tests from TensorpipeSerialize (6 ms total)
2022-11-23T02:20:47.1215014Z 
2022-11-23T02:20:47.1215242Z [----------] Global test environment tear-down
2022-11-23T02:20:47.1216055Z [==========] 8 tests from 3 test suites ran. (420 ms total)
2022-11-23T02:20:47.1216395Z [  PASSED  ] 8 tests.
2022-11-23T02:20:47.1216554Z 
2022-11-23T02:20:47.1216707Z   YOU HAVE 1 DISABLED TEST
2022-11-23T02:20:47.1216873Z 
2022-11-23T02:20:47.1921598Z 
2022-11-23T02:20:47.1964997Z Prepare all required actions
2022-11-23T02:20:47.1965293Z Getting action download info
2022-11-23T02:20:47.3495817Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T02:20:47.5652397Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T02:20:47.5652604Z with:
2022-11-23T02:20:47.5652937Z   github-token: ***
2022-11-23T02:20:47.5653106Z env:
2022-11-23T02:20:47.5653266Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:47.5653535Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:47.5653862Z ##[endgroup]
2022-11-23T02:20:47.5681604Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T02:20:47.5681835Z with:
2022-11-23T02:20:47.5681995Z   shell: bash
2022-11-23T02:20:47.5682159Z   timeout_minutes: 10
2022-11-23T02:20:47.5682338Z   max_attempts: 5
2022-11-23T02:20:47.5682575Z   retry_wait_seconds: 30
2022-11-23T02:20:47.5682951Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T02:20:47.5683338Z   polling_interval_seconds: 1
2022-11-23T02:20:47.5683533Z   warning_on_retry: true
2022-11-23T02:20:47.5683716Z   continue_on_error: false
2022-11-23T02:20:47.5683875Z env:
2022-11-23T02:20:47.5684041Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:47.5684308Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:47.5684672Z   GITHUB_TOKEN: ***
2022-11-23T02:20:47.5684844Z ##[endgroup]
2022-11-23T02:20:47.9062053Z + python3 -m pip install requests==2.26.0
2022-11-23T02:20:48.6324049Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:20:48.6591766Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-11-23T02:20:48.6724965Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-11-23T02:20:48.6744801Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-11-23T02:20:48.6760021Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12)
2022-11-23T02:20:48.6919303Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-11-23T02:20:48.8932626Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293562 i-038a48515f8572324
2022-11-23T02:20:51.4949475Z + GHA_WORKFLOW_JOB_ID=9655066953
2022-11-23T02:20:51.4951065Z + echo job-id=9655066953
2022-11-23T02:20:51.9068967Z Command completed after 1 attempt(s).
2022-11-23T02:20:51.9175788Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:20:51.9176027Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:20:51.9187535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:20:51.9187752Z env:
2022-11-23T02:20:51.9187927Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:51.9188186Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:51.9188454Z   MONITOR_SCRIPT_PID: 15505
2022-11-23T02:20:51.9188639Z ##[endgroup]
2022-11-23T02:20:51.9277273Z Prepare all required actions
2022-11-23T02:20:51.9277533Z Getting action download info
2022-11-23T02:20:52.1180665Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T02:20:52.2711565Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T02:20:52.2711789Z with:
2022-11-23T02:20:52.2712022Z   file-suffix: test-distributed-1-2-linux.2xlarge_9655066953
2022-11-23T02:20:52.2712236Z env:
2022-11-23T02:20:52.2712407Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:52.2712674Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:52.2712986Z ##[endgroup]
2022-11-23T02:20:52.2735814Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T02:20:52.2736096Z # Remove any previous test jsons if they exist
2022-11-23T02:20:52.2736325Z rm -f test-jsons-*.zip
2022-11-23T02:20:52.2736601Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T02:20:52.2747511Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:20:52.2747731Z env:
2022-11-23T02:20:52.2747903Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:52.2748159Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:52.2748479Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9655066953
2022-11-23T02:20:52.2748715Z ##[endgroup]
2022-11-23T02:20:52.2976459Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T02:20:52.3016550Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T02:20:52.3033870Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T02:20:52.3035306Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T02:20:52.3043722Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T02:20:52.3062293Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:20:52.3062586Z # Remove any previous test reports if they exist
2022-11-23T02:20:52.3062823Z rm -f test-reports-*.zip
2022-11-23T02:20:52.3063084Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T02:20:52.3073563Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:20:52.3073783Z env:
2022-11-23T02:20:52.3073962Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:52.3074215Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:52.3074535Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9655066953
2022-11-23T02:20:52.3074772Z ##[endgroup]
2022-11-23T02:20:52.3252772Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010406.xml (deflated 42%)
2022-11-23T02:20:52.3258176Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010410.xml (deflated 42%)
2022-11-23T02:20:52.3261988Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010413.xml (deflated 42%)
2022-11-23T02:20:52.3268754Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010415.xml (deflated 42%)
2022-11-23T02:20:52.3281677Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010416.xml (deflated 42%)
2022-11-23T02:20:52.3285950Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010417.xml (deflated 42%)
2022-11-23T02:20:52.3291703Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010420.xml (deflated 42%)
2022-11-23T02:20:52.3298076Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010424.xml (deflated 42%)
2022-11-23T02:20:52.3309477Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010427.xml (deflated 42%)
2022-11-23T02:20:52.3318166Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010429.xml (deflated 42%)
2022-11-23T02:20:52.3324439Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010430.xml (deflated 42%)
2022-11-23T02:20:52.3328416Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123010431.xml (deflated 42%)
2022-11-23T02:20:52.3337950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010438.xml (deflated 42%)
2022-11-23T02:20:52.3340669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010442.xml (deflated 42%)
2022-11-23T02:20:52.3343180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010443.xml (deflated 43%)
2022-11-23T02:20:52.3345753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010447.xml (deflated 42%)
2022-11-23T02:20:52.3352621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010449.xml (deflated 41%)
2022-11-23T02:20:52.3356408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010453.xml (deflated 41%)
2022-11-23T02:20:52.3359410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010457.xml (deflated 42%)
2022-11-23T02:20:52.3363117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010501.xml (deflated 41%)
2022-11-23T02:20:52.3375490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010505.xml (deflated 41%)
2022-11-23T02:20:52.3384537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010509.xml (deflated 41%)
2022-11-23T02:20:52.3385139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010513.xml (deflated 41%)
2022-11-23T02:20:52.3385721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010517.xml (deflated 42%)
2022-11-23T02:20:52.3386309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010521.xml (deflated 41%)
2022-11-23T02:20:52.3386903Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010525.xml (deflated 42%)
2022-11-23T02:20:52.3387490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010527.xml (deflated 41%)
2022-11-23T02:20:52.3388078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010531.xml (deflated 42%)
2022-11-23T02:20:52.3388668Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010533.xml (deflated 43%)
2022-11-23T02:20:52.3389238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010537.xml (deflated 42%)
2022-11-23T02:20:52.3389824Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010538.xml (deflated 46%)
2022-11-23T02:20:52.3390517Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010542.xml (deflated 47%)
2022-11-23T02:20:52.3391111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010546.xml (deflated 48%)
2022-11-23T02:20:52.3391683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010550.xml (deflated 46%)
2022-11-23T02:20:52.3392264Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010554.xml (deflated 41%)
2022-11-23T02:20:52.3392853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010558.xml (deflated 41%)
2022-11-23T02:20:52.3393484Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010602.xml (deflated 41%)
2022-11-23T02:20:52.3394065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010606.xml (deflated 41%)
2022-11-23T02:20:52.3394665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010610.xml (deflated 41%)
2022-11-23T02:20:52.3395248Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010614.xml (deflated 41%)
2022-11-23T02:20:52.3395833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010618.xml (deflated 41%)
2022-11-23T02:20:52.3396408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010622.xml (deflated 43%)
2022-11-23T02:20:52.3396998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010623.xml (deflated 42%)
2022-11-23T02:20:52.3397581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010625.xml (deflated 41%)
2022-11-23T02:20:52.3398165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010629.xml (deflated 41%)
2022-11-23T02:20:52.3398739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010633.xml (deflated 43%)
2022-11-23T02:20:52.3399322Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010635.xml (deflated 43%)
2022-11-23T02:20:52.3399902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010636.xml (deflated 43%)
2022-11-23T02:20:52.3400493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010638.xml (deflated 43%)
2022-11-23T02:20:52.3401073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010639.xml (deflated 40%)
2022-11-23T02:20:52.3401663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010643.xml (deflated 40%)
2022-11-23T02:20:52.3402244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010647.xml (deflated 43%)
2022-11-23T02:20:52.3402831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010649.xml (deflated 41%)
2022-11-23T02:20:52.3403418Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010653.xml (deflated 41%)
2022-11-23T02:20:52.3403998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010657.xml (deflated 41%)
2022-11-23T02:20:52.3404577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010701.xml (deflated 41%)
2022-11-23T02:20:52.3405293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010705.xml (deflated 41%)
2022-11-23T02:20:52.3405879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010709.xml (deflated 41%)
2022-11-23T02:20:52.3406453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010713.xml (deflated 41%)
2022-11-23T02:20:52.3407042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010717.xml (deflated 41%)
2022-11-23T02:20:52.3407620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010721.xml (deflated 41%)
2022-11-23T02:20:52.3408237Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010725.xml (deflated 41%)
2022-11-23T02:20:52.3408809Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010729.xml (deflated 41%)
2022-11-23T02:20:52.3409394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010733.xml (deflated 41%)
2022-11-23T02:20:52.3409979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010737.xml (deflated 41%)
2022-11-23T02:20:52.3410565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010741.xml (deflated 41%)
2022-11-23T02:20:52.3411134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010745.xml (deflated 40%)
2022-11-23T02:20:52.3411720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010749.xml (deflated 41%)
2022-11-23T02:20:52.3412303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010753.xml (deflated 41%)
2022-11-23T02:20:52.3412885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010758.xml (deflated 40%)
2022-11-23T02:20:52.3413457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010801.xml (deflated 41%)
2022-11-23T02:20:52.3414073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010805.xml (deflated 40%)
2022-11-23T02:20:52.3414655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010809.xml (deflated 41%)
2022-11-23T02:20:52.3415408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010813.xml (deflated 41%)
2022-11-23T02:20:52.3415989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010817.xml (deflated 41%)
2022-11-23T02:20:52.3416560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010821.xml (deflated 41%)
2022-11-23T02:20:52.3417147Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010825.xml (deflated 42%)
2022-11-23T02:20:52.3417727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010829.xml (deflated 41%)
2022-11-23T02:20:52.3418309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010833.xml (deflated 41%)
2022-11-23T02:20:52.3418880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010837.xml (deflated 42%)
2022-11-23T02:20:52.3419464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010841.xml (deflated 41%)
2022-11-23T02:20:52.3420106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010845.xml (deflated 41%)
2022-11-23T02:20:52.3420695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010849.xml (deflated 41%)
2022-11-23T02:20:52.3421266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010853.xml (deflated 42%)
2022-11-23T02:20:52.3421847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010857.xml (deflated 42%)
2022-11-23T02:20:52.3422474Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010901.xml (deflated 42%)
2022-11-23T02:20:52.3423056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010905.xml (deflated 43%)
2022-11-23T02:20:52.3423632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010906.xml (deflated 43%)
2022-11-23T02:20:52.3424216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010908.xml (deflated 43%)
2022-11-23T02:20:52.3424799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010910.xml (deflated 42%)
2022-11-23T02:20:52.3425380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010911.xml (deflated 43%)
2022-11-23T02:20:52.3425949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010913.xml (deflated 42%)
2022-11-23T02:20:52.3426531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010914.xml (deflated 43%)
2022-11-23T02:20:52.3427114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010916.xml (deflated 43%)
2022-11-23T02:20:52.3427701Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010918.xml (deflated 43%)
2022-11-23T02:20:52.3428281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010919.xml (deflated 43%)
2022-11-23T02:20:52.3428849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010921.xml (deflated 43%)
2022-11-23T02:20:52.3429430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010923.xml (deflated 43%)
2022-11-23T02:20:52.3430033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010924.xml (deflated 43%)
2022-11-23T02:20:52.3430623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010926.xml (deflated 43%)
2022-11-23T02:20:52.3431186Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010927.xml (deflated 43%)
2022-11-23T02:20:52.3431768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010929.xml (deflated 43%)
2022-11-23T02:20:52.3432353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010931.xml (deflated 43%)
2022-11-23T02:20:52.3432935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010932.xml (deflated 43%)
2022-11-23T02:20:52.3433516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010934.xml (deflated 43%)
2022-11-23T02:20:52.3434136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010935.xml (deflated 42%)
2022-11-23T02:20:52.3434754Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010937.xml (deflated 43%)
2022-11-23T02:20:52.3435337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010939.xml (deflated 43%)
2022-11-23T02:20:52.3435918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010940.xml (deflated 43%)
2022-11-23T02:20:52.3436491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010942.xml (deflated 43%)
2022-11-23T02:20:52.3437105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010943.xml (deflated 42%)
2022-11-23T02:20:52.3437693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010947.xml (deflated 42%)
2022-11-23T02:20:52.3438280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010949.xml (deflated 42%)
2022-11-23T02:20:52.3438851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010950.xml (deflated 41%)
2022-11-23T02:20:52.3439434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123010956.xml (deflated 42%)
2022-11-23T02:20:52.3440021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011000.xml (deflated 41%)
2022-11-23T02:20:52.3440603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011005.xml (deflated 42%)
2022-11-23T02:20:52.3441174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011009.xml (deflated 41%)
2022-11-23T02:20:52.3441760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011014.xml (deflated 42%)
2022-11-23T02:20:52.3442341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011017.xml (deflated 41%)
2022-11-23T02:20:52.3442920Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011023.xml (deflated 42%)
2022-11-23T02:20:52.3443493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011024.xml (deflated 41%)
2022-11-23T02:20:52.3444075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011033.xml (deflated 41%)
2022-11-23T02:20:52.3444657Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011037.xml (deflated 40%)
2022-11-23T02:20:52.3445318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011041.xml (deflated 42%)
2022-11-23T02:20:52.3445890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011043.xml (deflated 42%)
2022-11-23T02:20:52.3446471Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011044.xml (deflated 41%)
2022-11-23T02:20:52.3447050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011046.xml (deflated 41%)
2022-11-23T02:20:52.3447632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011047.xml (deflated 42%)
2022-11-23T02:20:52.3448199Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011049.xml (deflated 41%)
2022-11-23T02:20:52.3448824Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011051.xml (deflated 42%)
2022-11-23T02:20:52.3449403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011052.xml (deflated 42%)
2022-11-23T02:20:52.3449983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011054.xml (deflated 41%)
2022-11-23T02:20:52.3450546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011058.xml (deflated 42%)
2022-11-23T02:20:52.3451127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011100.xml (deflated 41%)
2022-11-23T02:20:52.3451740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011104.xml (deflated 41%)
2022-11-23T02:20:52.3452321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011108.xml (deflated 42%)
2022-11-23T02:20:52.3452893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011112.xml (deflated 42%)
2022-11-23T02:20:52.3453476Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011114.xml (deflated 41%)
2022-11-23T02:20:52.3454055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011115.xml (deflated 41%)
2022-11-23T02:20:52.3454635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011117.xml (deflated 42%)
2022-11-23T02:20:52.3455340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011121.xml (deflated 41%)
2022-11-23T02:20:52.3455927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011125.xml (deflated 42%)
2022-11-23T02:20:52.3456511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011127.xml (deflated 42%)
2022-11-23T02:20:52.3457096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011129.xml (deflated 42%)
2022-11-23T02:20:52.3457667Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011130.xml (deflated 42%)
2022-11-23T02:20:52.3458253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011134.xml (deflated 42%)
2022-11-23T02:20:52.3458834Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011138.xml (deflated 41%)
2022-11-23T02:20:52.3459419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011140.xml (deflated 41%)
2022-11-23T02:20:52.3459991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011141.xml (deflated 41%)
2022-11-23T02:20:52.3460631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011145.xml (deflated 42%)
2022-11-23T02:20:52.3461211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011147.xml (deflated 42%)
2022-11-23T02:20:52.3461796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011148.xml (deflated 42%)
2022-11-23T02:20:52.3462369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011150.xml (deflated 42%)
2022-11-23T02:20:52.3462956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011152.xml (deflated 42%)
2022-11-23T02:20:52.3463605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011156.xml (deflated 41%)
2022-11-23T02:20:52.3464192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011200.xml (deflated 42%)
2022-11-23T02:20:52.3464763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011202.xml (deflated 41%)
2022-11-23T02:20:52.3465346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011206.xml (deflated 42%)
2022-11-23T02:20:52.3465983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011210.xml (deflated 42%)
2022-11-23T02:20:52.3466567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011213.xml (deflated 42%)
2022-11-23T02:20:52.3467142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011217.xml (deflated 42%)
2022-11-23T02:20:52.3467793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011221.xml (deflated 42%)
2022-11-23T02:20:52.3468362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011225.xml (deflated 42%)
2022-11-23T02:20:52.3468944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011229.xml (deflated 42%)
2022-11-23T02:20:52.3469530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011233.xml (deflated 42%)
2022-11-23T02:20:52.3470115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011237.xml (deflated 42%)
2022-11-23T02:20:52.3470691Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011240.xml (deflated 42%)
2022-11-23T02:20:52.3471279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011244.xml (deflated 42%)
2022-11-23T02:20:52.3471863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011248.xml (deflated 42%)
2022-11-23T02:20:52.3472448Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011252.xml (deflated 41%)
2022-11-23T02:20:52.3473015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011254.xml (deflated 42%)
2022-11-23T02:20:52.3473600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011258.xml (deflated 41%)
2022-11-23T02:20:52.3474187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011301.xml (deflated 41%)
2022-11-23T02:20:52.3474773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011305.xml (deflated 42%)
2022-11-23T02:20:52.3475344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011309.xml (deflated 41%)
2022-11-23T02:20:52.3475931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011311.xml (deflated 41%)
2022-11-23T02:20:52.3476511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011313.xml (deflated 42%)
2022-11-23T02:20:52.3477100Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011314.xml (deflated 42%)
2022-11-23T02:20:52.3477697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011316.xml (deflated 42%)
2022-11-23T02:20:52.3478280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011317.xml (deflated 42%)
2022-11-23T02:20:52.3478869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011321.xml (deflated 42%)
2022-11-23T02:20:52.3479450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011323.xml (deflated 42%)
2022-11-23T02:20:52.3480022Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011325.xml (deflated 42%)
2022-11-23T02:20:52.3480634Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011326.xml (deflated 42%)
2022-11-23T02:20:52.3481216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011330.xml (deflated 42%)
2022-11-23T02:20:52.3481805Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011332.xml (deflated 41%)
2022-11-23T02:20:52.3482374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011334.xml (deflated 42%)
2022-11-23T02:20:52.3482962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011336.xml (deflated 41%)
2022-11-23T02:20:52.3483547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011340.xml (deflated 42%)
2022-11-23T02:20:52.3484133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011344.xml (deflated 42%)
2022-11-23T02:20:52.3484709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011348.xml (deflated 42%)
2022-11-23T02:20:52.3485399Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011349.xml (deflated 42%)
2022-11-23T02:20:52.3485986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011351.xml (deflated 42%)
2022-11-23T02:20:52.3486568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011353.xml (deflated 41%)
2022-11-23T02:20:52.3487141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011355.xml (deflated 42%)
2022-11-23T02:20:52.3487729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011359.xml (deflated 41%)
2022-11-23T02:20:52.3488313Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011403.xml (deflated 41%)
2022-11-23T02:20:52.3488894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011407.xml (deflated 42%)
2022-11-23T02:20:52.3489463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011409.xml (deflated 41%)
2022-11-23T02:20:52.3490048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011411.xml (deflated 41%)
2022-11-23T02:20:52.3490635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011414.xml (deflated 41%)
2022-11-23T02:20:52.3491220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011419.xml (deflated 41%)
2022-11-23T02:20:52.3491789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011422.xml (deflated 43%)
2022-11-23T02:20:52.3492408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011424.xml (deflated 41%)
2022-11-23T02:20:52.3492989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011428.xml (deflated 41%)
2022-11-23T02:20:52.3493573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011432.xml (deflated 41%)
2022-11-23T02:20:52.3494154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011436.xml (deflated 42%)
2022-11-23T02:20:52.3494717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011438.xml (deflated 41%)
2022-11-23T02:20:52.3495458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011442.xml (deflated 41%)
2022-11-23T02:20:52.3496044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011446.xml (deflated 41%)
2022-11-23T02:20:52.3496626Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011450.xml (deflated 40%)
2022-11-23T02:20:52.3497197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011454.xml (deflated 40%)
2022-11-23T02:20:52.3497784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011458.xml (deflated 41%)
2022-11-23T02:20:52.3498368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011459.xml (deflated 41%)
2022-11-23T02:20:52.3498956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011503.xml (deflated 41%)
2022-11-23T02:20:52.3499527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011507.xml (deflated 41%)
2022-11-23T02:20:52.3500114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011511.xml (deflated 41%)
2022-11-23T02:20:52.3500702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011515.xml (deflated 41%)
2022-11-23T02:20:52.3501281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011517.xml (deflated 41%)
2022-11-23T02:20:52.3501848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011519.xml (deflated 41%)
2022-11-23T02:20:52.3502435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011525.xml (deflated 41%)
2022-11-23T02:20:52.3503021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011531.xml (deflated 41%)
2022-11-23T02:20:52.3503607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011535.xml (deflated 40%)
2022-11-23T02:20:52.3504185Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011539.xml (deflated 40%)
2022-11-23T02:20:52.3504762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011543.xml (deflated 42%)
2022-11-23T02:20:52.3505346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011545.xml (deflated 42%)
2022-11-23T02:20:52.3505927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011546.xml (deflated 42%)
2022-11-23T02:20:52.3506500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011548.xml (deflated 42%)
2022-11-23T02:20:52.3507145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011550.xml (deflated 43%)
2022-11-23T02:20:52.3507731Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011551.xml (deflated 42%)
2022-11-23T02:20:52.3508320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011553.xml (deflated 42%)
2022-11-23T02:20:52.3508893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011554.xml (deflated 42%)
2022-11-23T02:20:52.3509518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011556.xml (deflated 41%)
2022-11-23T02:20:52.3510108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011600.xml (deflated 42%)
2022-11-23T02:20:52.3510692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011604.xml (deflated 42%)
2022-11-23T02:20:52.3511260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011605.xml (deflated 43%)
2022-11-23T02:20:52.3511841Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011607.xml (deflated 42%)
2022-11-23T02:20:52.3512425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011609.xml (deflated 42%)
2022-11-23T02:20:52.3513005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011613.xml (deflated 42%)
2022-11-23T02:20:52.3513576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011617.xml (deflated 42%)
2022-11-23T02:20:52.3514161Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011621.xml (deflated 42%)
2022-11-23T02:20:52.3514739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011622.xml (deflated 42%)
2022-11-23T02:20:52.3515323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011624.xml (deflated 41%)
2022-11-23T02:20:52.3515894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011628.xml (deflated 41%)
2022-11-23T02:20:52.3516482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011632.xml (deflated 42%)
2022-11-23T02:20:52.3517068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011634.xml (deflated 41%)
2022-11-23T02:20:52.3517655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011638.xml (deflated 41%)
2022-11-23T02:20:52.3518226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011642.xml (deflated 41%)
2022-11-23T02:20:52.3518803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011646.xml (deflated 41%)
2022-11-23T02:20:52.3519385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011650.xml (deflated 41%)
2022-11-23T02:20:52.3519970Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011654.xml (deflated 41%)
2022-11-23T02:20:52.3520555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011658.xml (deflated 41%)
2022-11-23T02:20:52.3521246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011702.xml (deflated 41%)
2022-11-23T02:20:52.3521832Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011706.xml (deflated 41%)
2022-11-23T02:20:52.3522397Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011710.xml (deflated 41%)
2022-11-23T02:20:52.3522980Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011714.xml (deflated 43%)
2022-11-23T02:20:52.3523567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011716.xml (deflated 42%)
2022-11-23T02:20:52.3524177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011720.xml (deflated 44%)
2022-11-23T02:20:52.3524746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011722.xml (deflated 43%)
2022-11-23T02:20:52.3525404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011723.xml (deflated 41%)
2022-11-23T02:20:52.3525987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011727.xml (deflated 43%)
2022-11-23T02:20:52.3526570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011729.xml (deflated 43%)
2022-11-23T02:20:52.3527139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011730.xml (deflated 42%)
2022-11-23T02:20:52.3527725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011734.xml (deflated 41%)
2022-11-23T02:20:52.3528302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011738.xml (deflated 41%)
2022-11-23T02:20:52.3528886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011742.xml (deflated 41%)
2022-11-23T02:20:52.3529456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011746.xml (deflated 42%)
2022-11-23T02:20:52.3530044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011748.xml (deflated 41%)
2022-11-23T02:20:52.3530628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011750.xml (deflated 41%)
2022-11-23T02:20:52.3531215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011754.xml (deflated 41%)
2022-11-23T02:20:52.3531779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011758.xml (deflated 41%)
2022-11-23T02:20:52.3532360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011801.xml (deflated 41%)
2022-11-23T02:20:52.3532949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011805.xml (deflated 41%)
2022-11-23T02:20:52.3533529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011809.xml (deflated 41%)
2022-11-23T02:20:52.3534101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011813.xml (deflated 41%)
2022-11-23T02:20:52.3534686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011817.xml (deflated 41%)
2022-11-23T02:20:52.3535389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011821.xml (deflated 42%)
2022-11-23T02:20:52.3536036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011823.xml (deflated 42%)
2022-11-23T02:20:52.3536607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011824.xml (deflated 42%)
2022-11-23T02:20:52.3537192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011826.xml (deflated 41%)
2022-11-23T02:20:52.3537776Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011830.xml (deflated 41%)
2022-11-23T02:20:52.3538362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011834.xml (deflated 41%)
2022-11-23T02:20:52.3538975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011838.xml (deflated 41%)
2022-11-23T02:20:52.3539562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011842.xml (deflated 41%)
2022-11-23T02:20:52.3540145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011846.xml (deflated 42%)
2022-11-23T02:20:52.3540728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011849.xml (deflated 42%)
2022-11-23T02:20:52.3541307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011853.xml (deflated 40%)
2022-11-23T02:20:52.3541887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011857.xml (deflated 42%)
2022-11-23T02:20:52.3542469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011901.xml (deflated 42%)
2022-11-23T02:20:52.3543058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011905.xml (deflated 41%)
2022-11-23T02:20:52.3543637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011907.xml (deflated 41%)
2022-11-23T02:20:52.3544211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011910.xml (deflated 43%)
2022-11-23T02:20:52.3544797Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011914.xml (deflated 42%)
2022-11-23T02:20:52.3545381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011915.xml (deflated 43%)
2022-11-23T02:20:52.3545965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011919.xml (deflated 42%)
2022-11-23T02:20:52.3546541Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011921.xml (deflated 42%)
2022-11-23T02:20:52.3547126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011925.xml (deflated 41%)
2022-11-23T02:20:52.3547709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011929.xml (deflated 42%)
2022-11-23T02:20:52.3548290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011933.xml (deflated 41%)
2022-11-23T02:20:52.3548860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011937.xml (deflated 42%)
2022-11-23T02:20:52.3549446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011941.xml (deflated 41%)
2022-11-23T02:20:52.3550053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011945.xml (deflated 41%)
2022-11-23T02:20:52.3550716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011949.xml (deflated 41%)
2022-11-23T02:20:52.3551316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011953.xml (deflated 41%)
2022-11-23T02:20:52.3551908Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011956.xml (deflated 42%)
2022-11-23T02:20:52.3552518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123011958.xml (deflated 42%)
2022-11-23T02:20:52.3553131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012002.xml (deflated 42%)
2022-11-23T02:20:52.3553719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012004.xml (deflated 43%)
2022-11-23T02:20:52.3554284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012008.xml (deflated 42%)
2022-11-23T02:20:52.3554872Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012010.xml (deflated 46%)
2022-11-23T02:20:52.3555451Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012014.xml (deflated 47%)
2022-11-23T02:20:52.3556035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012018.xml (deflated 49%)
2022-11-23T02:20:52.3556608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012021.xml (deflated 46%)
2022-11-23T02:20:52.3557191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012025.xml (deflated 41%)
2022-11-23T02:20:52.3557782Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012029.xml (deflated 41%)
2022-11-23T02:20:52.3558360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012033.xml (deflated 41%)
2022-11-23T02:20:52.3558928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012037.xml (deflated 41%)
2022-11-23T02:20:52.3559515Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012041.xml (deflated 41%)
2022-11-23T02:20:52.3560096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012045.xml (deflated 41%)
2022-11-23T02:20:52.3560684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012049.xml (deflated 40%)
2022-11-23T02:20:52.3561252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012053.xml (deflated 43%)
2022-11-23T02:20:52.3561837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012055.xml (deflated 42%)
2022-11-23T02:20:52.3562417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012057.xml (deflated 41%)
2022-11-23T02:20:52.3563000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012101.xml (deflated 41%)
2022-11-23T02:20:52.3563571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012105.xml (deflated 43%)
2022-11-23T02:20:52.3564154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012106.xml (deflated 43%)
2022-11-23T02:20:52.3564773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012108.xml (deflated 43%)
2022-11-23T02:20:52.3565438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012110.xml (deflated 43%)
2022-11-23T02:20:52.3566015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012111.xml (deflated 41%)
2022-11-23T02:20:52.3566602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012115.xml (deflated 40%)
2022-11-23T02:20:52.3567181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012119.xml (deflated 43%)
2022-11-23T02:20:52.3567821Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012121.xml (deflated 41%)
2022-11-23T02:20:52.3568401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012125.xml (deflated 41%)
2022-11-23T02:20:52.3568978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012129.xml (deflated 41%)
2022-11-23T02:20:52.3569558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012133.xml (deflated 41%)
2022-11-23T02:20:52.3570540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012137.xml (deflated 41%)
2022-11-23T02:20:52.3571132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012141.xml (deflated 41%)
2022-11-23T02:20:52.3571712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012145.xml (deflated 41%)
2022-11-23T02:20:52.3572294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012149.xml (deflated 41%)
2022-11-23T02:20:52.3572885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012153.xml (deflated 41%)
2022-11-23T02:20:52.3573463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012157.xml (deflated 41%)
2022-11-23T02:20:52.3574035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012201.xml (deflated 41%)
2022-11-23T02:20:52.3574620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012205.xml (deflated 41%)
2022-11-23T02:20:52.3575324Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012209.xml (deflated 42%)
2022-11-23T02:20:52.3575911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012213.xml (deflated 42%)
2022-11-23T02:20:52.3576490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012217.xml (deflated 41%)
2022-11-23T02:20:52.3577068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012221.xml (deflated 41%)
2022-11-23T02:20:52.3577650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012225.xml (deflated 41%)
2022-11-23T02:20:52.3578230Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012229.xml (deflated 41%)
2022-11-23T02:20:52.3578808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012233.xml (deflated 41%)
2022-11-23T02:20:52.3579394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012237.xml (deflated 41%)
2022-11-23T02:20:52.3580049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012241.xml (deflated 41%)
2022-11-23T02:20:52.3580638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012246.xml (deflated 41%)
2022-11-23T02:20:52.3581208Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012250.xml (deflated 41%)
2022-11-23T02:20:52.3581788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012254.xml (deflated 41%)
2022-11-23T02:20:52.3582369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml (deflated 42%)
2022-11-23T02:20:52.3582998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012301.xml (deflated 42%)
2022-11-23T02:20:52.3583576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012305.xml (deflated 41%)
2022-11-23T02:20:52.3584155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012309.xml (deflated 42%)
2022-11-23T02:20:52.3584738Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012313.xml (deflated 41%)
2022-11-23T02:20:52.3585325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012317.xml (deflated 41%)
2022-11-23T02:20:52.3585896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml (deflated 41%)
2022-11-23T02:20:52.3586479Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012325.xml (deflated 42%)
2022-11-23T02:20:52.3587070Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012329.xml (deflated 42%)
2022-11-23T02:20:52.3587656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012333.xml (deflated 41%)
2022-11-23T02:20:52.3588227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012337.xml (deflated 43%)
2022-11-23T02:20:52.3588811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012339.xml (deflated 43%)
2022-11-23T02:20:52.3589397Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012340.xml (deflated 43%)
2022-11-23T02:20:52.3589982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012342.xml (deflated 42%)
2022-11-23T02:20:52.3590551Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012343.xml (deflated 43%)
2022-11-23T02:20:52.3591135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012345.xml (deflated 42%)
2022-11-23T02:20:52.3591721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012347.xml (deflated 43%)
2022-11-23T02:20:52.3592306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012348.xml (deflated 43%)
2022-11-23T02:20:52.3592877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012350.xml (deflated 43%)
2022-11-23T02:20:52.3593462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012351.xml (deflated 43%)
2022-11-23T02:20:52.3594047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml (deflated 43%)
2022-11-23T02:20:52.3594663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012355.xml (deflated 43%)
2022-11-23T02:20:52.3595233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012356.xml (deflated 43%)
2022-11-23T02:20:52.3595818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012358.xml (deflated 43%)
2022-11-23T02:20:52.3596405Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012359.xml (deflated 43%)
2022-11-23T02:20:52.3597015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012401.xml (deflated 43%)
2022-11-23T02:20:52.3597588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012403.xml (deflated 43%)
2022-11-23T02:20:52.3598171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012404.xml (deflated 42%)
2022-11-23T02:20:52.3598746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012406.xml (deflated 43%)
2022-11-23T02:20:52.3599329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012407.xml (deflated 43%)
2022-11-23T02:20:52.3599909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012409.xml (deflated 43%)
2022-11-23T02:20:52.3600480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012411.xml (deflated 43%)
2022-11-23T02:20:52.3601066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012412.xml (deflated 43%)
2022-11-23T02:20:52.3601648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012414.xml (deflated 43%)
2022-11-23T02:20:52.3602234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml (deflated 42%)
2022-11-23T02:20:52.3602804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012419.xml (deflated 42%)
2022-11-23T02:20:52.3603387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012421.xml (deflated 42%)
2022-11-23T02:20:52.3603969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012423.xml (deflated 41%)
2022-11-23T02:20:52.3604555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012428.xml (deflated 42%)
2022-11-23T02:20:52.3605201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012432.xml (deflated 40%)
2022-11-23T02:20:52.3605787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml (deflated 41%)
2022-11-23T02:20:52.3606364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012441.xml (deflated 41%)
2022-11-23T02:20:52.3606945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012446.xml (deflated 42%)
2022-11-23T02:20:52.3607516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml (deflated 41%)
2022-11-23T02:20:52.3608097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012455.xml (deflated 41%)
2022-11-23T02:20:52.3608733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012500.xml (deflated 41%)
2022-11-23T02:20:52.3609312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012509.xml (deflated 41%)
2022-11-23T02:20:52.3609884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012513.xml (deflated 40%)
2022-11-23T02:20:52.3610469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012516.xml (deflated 42%)
2022-11-23T02:20:52.3611055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012518.xml (deflated 42%)
2022-11-23T02:20:52.3611749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012520.xml (deflated 42%)
2022-11-23T02:20:52.3612431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012521.xml (deflated 41%)
2022-11-23T02:20:52.3613018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012523.xml (deflated 42%)
2022-11-23T02:20:52.3613587Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml (deflated 42%)
2022-11-23T02:20:52.3614174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012526.xml (deflated 41%)
2022-11-23T02:20:52.3614763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012528.xml (deflated 42%)
2022-11-23T02:20:52.3615472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012529.xml (deflated 41%)
2022-11-23T02:20:52.3616045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012533.xml (deflated 42%)
2022-11-23T02:20:52.3616633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012535.xml (deflated 40%)
2022-11-23T02:20:52.3617214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012539.xml (deflated 41%)
2022-11-23T02:20:52.3617795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012543.xml (deflated 42%)
2022-11-23T02:20:52.3618367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012547.xml (deflated 42%)
2022-11-23T02:20:52.3618954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012549.xml (deflated 41%)
2022-11-23T02:20:52.3619533Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml (deflated 41%)
2022-11-23T02:20:52.3620121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012552.xml (deflated 42%)
2022-11-23T02:20:52.3620690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012556.xml (deflated 41%)
2022-11-23T02:20:52.3621273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012600.xml (deflated 42%)
2022-11-23T02:20:52.3621855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml (deflated 42%)
2022-11-23T02:20:52.3622436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012604.xml (deflated 41%)
2022-11-23T02:20:52.3623012Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012605.xml (deflated 41%)
2022-11-23T02:20:52.3623659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012609.xml (deflated 41%)
2022-11-23T02:20:52.3624244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012613.xml (deflated 41%)
2022-11-23T02:20:52.3624890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012615.xml (deflated 41%)
2022-11-23T02:20:52.3625477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012616.xml (deflated 41%)
2022-11-23T02:20:52.3626045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012620.xml (deflated 42%)
2022-11-23T02:20:52.3626689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012622.xml (deflated 42%)
2022-11-23T02:20:52.3627273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012624.xml (deflated 42%)
2022-11-23T02:20:52.3627855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml (deflated 42%)
2022-11-23T02:20:52.3628420Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012627.xml (deflated 41%)
2022-11-23T02:20:52.3628999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012631.xml (deflated 41%)
2022-11-23T02:20:52.3629581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012635.xml (deflated 42%)
2022-11-23T02:20:52.3630163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012637.xml (deflated 41%)
2022-11-23T02:20:52.3630737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012641.xml (deflated 42%)
2022-11-23T02:20:52.3631324Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012645.xml (deflated 42%)
2022-11-23T02:20:52.3631901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012649.xml (deflated 42%)
2022-11-23T02:20:52.3632485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012653.xml (deflated 42%)
2022-11-23T02:20:52.3633053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012657.xml (deflated 41%)
2022-11-23T02:20:52.3633641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012700.xml (deflated 42%)
2022-11-23T02:20:52.3634220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012704.xml (deflated 42%)
2022-11-23T02:20:52.3634804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012708.xml (deflated 42%)
2022-11-23T02:20:52.3635370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012712.xml (deflated 42%)
2022-11-23T02:20:52.3635949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012716.xml (deflated 42%)
2022-11-23T02:20:52.3636531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012720.xml (deflated 42%)
2022-11-23T02:20:52.3637116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012724.xml (deflated 42%)
2022-11-23T02:20:52.3637686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012728.xml (deflated 41%)
2022-11-23T02:20:52.3638302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml (deflated 42%)
2022-11-23T02:20:52.3638886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012733.xml (deflated 42%)
2022-11-23T02:20:52.3639466Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012737.xml (deflated 40%)
2022-11-23T02:20:52.3640037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012741.xml (deflated 42%)
2022-11-23T02:20:52.3640648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012745.xml (deflated 41%)
2022-11-23T02:20:52.3641225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012747.xml (deflated 41%)
2022-11-23T02:20:52.3641807Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012748.xml (deflated 42%)
2022-11-23T02:20:52.3642378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml (deflated 42%)
2022-11-23T02:20:52.3642962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012751.xml (deflated 42%)
2022-11-23T02:20:52.3643546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012753.xml (deflated 42%)
2022-11-23T02:20:52.3644128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml (deflated 42%)
2022-11-23T02:20:52.3644699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012759.xml (deflated 42%)
2022-11-23T02:20:52.3645355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012800.xml (deflated 42%)
2022-11-23T02:20:52.3645939Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012802.xml (deflated 42%)
2022-11-23T02:20:52.3646576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012806.xml (deflated 42%)
2022-11-23T02:20:52.3647145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012808.xml (deflated 41%)
2022-11-23T02:20:52.3647730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012809.xml (deflated 42%)
2022-11-23T02:20:52.3648307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012811.xml (deflated 42%)
2022-11-23T02:20:52.3648894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml (deflated 42%)
2022-11-23T02:20:52.3649459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012819.xml (deflated 42%)
2022-11-23T02:20:52.3650045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012823.xml (deflated 42%)
2022-11-23T02:20:52.3650720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012825.xml (deflated 42%)
2022-11-23T02:20:52.3651375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml (deflated 42%)
2022-11-23T02:20:52.3651988Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012829.xml (deflated 41%)
2022-11-23T02:20:52.3652640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012830.xml (deflated 42%)
2022-11-23T02:20:52.3653332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml (deflated 41%)
2022-11-23T02:20:52.3653989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml (deflated 41%)
2022-11-23T02:20:52.3654631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012842.xml (deflated 42%)
2022-11-23T02:20:52.3655394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml (deflated 41%)
2022-11-23T02:20:52.3656126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012846.xml (deflated 40%)
2022-11-23T02:20:52.3656756Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012850.xml (deflated 41%)
2022-11-23T02:20:52.3657403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012854.xml (deflated 41%)
2022-11-23T02:20:52.3658038Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml (deflated 43%)
2022-11-23T02:20:52.3658687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012859.xml (deflated 41%)
2022-11-23T02:20:52.3659345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012903.xml (deflated 41%)
2022-11-23T02:20:52.3659991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012907.xml (deflated 40%)
2022-11-23T02:20:52.3660629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012911.xml (deflated 42%)
2022-11-23T02:20:52.3661305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012913.xml (deflated 41%)
2022-11-23T02:20:52.3661945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012917.xml (deflated 41%)
2022-11-23T02:20:52.3662573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012921.xml (deflated 41%)
2022-11-23T02:20:52.3663203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012925.xml (deflated 40%)
2022-11-23T02:20:52.3663857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012929.xml (deflated 41%)
2022-11-23T02:20:52.3664512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012933.xml (deflated 41%)
2022-11-23T02:20:52.3665158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012935.xml (deflated 41%)
2022-11-23T02:20:52.3665775Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012939.xml (deflated 41%)
2022-11-23T02:20:52.3666411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012943.xml (deflated 41%)
2022-11-23T02:20:52.3667060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012947.xml (deflated 41%)
2022-11-23T02:20:52.3667722Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012951.xml (deflated 41%)
2022-11-23T02:20:52.3668363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012952.xml (deflated 41%)
2022-11-23T02:20:52.3669065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012954.xml (deflated 41%)
2022-11-23T02:20:52.3669720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013000.xml (deflated 41%)
2022-11-23T02:20:52.3670362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013006.xml (deflated 41%)
2022-11-23T02:20:52.3670999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013010.xml (deflated 41%)
2022-11-23T02:20:52.3671633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013014.xml (deflated 41%)
2022-11-23T02:20:52.3672321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013019.xml (deflated 42%)
2022-11-23T02:20:52.3673021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013020.xml (deflated 43%)
2022-11-23T02:20:52.3673659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013022.xml (deflated 42%)
2022-11-23T02:20:52.3674319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013023.xml (deflated 42%)
2022-11-23T02:20:52.3674969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013025.xml (deflated 43%)
2022-11-23T02:20:52.3675617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013027.xml (deflated 42%)
2022-11-23T02:20:52.3676258Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013028.xml (deflated 42%)
2022-11-23T02:20:52.3676908Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013030.xml (deflated 42%)
2022-11-23T02:20:52.3677499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013031.xml (deflated 41%)
2022-11-23T02:20:52.3678152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013035.xml (deflated 42%)
2022-11-23T02:20:52.3678734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013039.xml (deflated 42%)
2022-11-23T02:20:52.3679404Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013041.xml (deflated 43%)
2022-11-23T02:20:52.3680049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013042.xml (deflated 42%)
2022-11-23T02:20:52.3680685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml (deflated 42%)
2022-11-23T02:20:52.3681306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013048.xml (deflated 42%)
2022-11-23T02:20:52.3681967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml (deflated 41%)
2022-11-23T02:20:52.3682621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013056.xml (deflated 42%)
2022-11-23T02:20:52.3683266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml (deflated 42%)
2022-11-23T02:20:52.3683907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml (deflated 41%)
2022-11-23T02:20:52.3684561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013104.xml (deflated 41%)
2022-11-23T02:20:52.3685290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013108.xml (deflated 42%)
2022-11-23T02:20:52.3685938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013110.xml (deflated 41%)
2022-11-23T02:20:52.3686576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013114.xml (deflated 41%)
2022-11-23T02:20:52.3687226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013118.xml (deflated 41%)
2022-11-23T02:20:52.3687951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml (deflated 41%)
2022-11-23T02:20:52.3688610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013126.xml (deflated 41%)
2022-11-23T02:20:52.3689264Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013130.xml (deflated 41%)
2022-11-23T02:20:52.3689901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013134.xml (deflated 41%)
2022-11-23T02:20:52.3690548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013138.xml (deflated 41%)
2022-11-23T02:20:52.3691194Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013142.xml (deflated 41%)
2022-11-23T02:20:52.3691823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013146.xml (deflated 41%)
2022-11-23T02:20:52.3692464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013150.xml (deflated 43%)
2022-11-23T02:20:52.3693115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013152.xml (deflated 41%)
2022-11-23T02:20:52.3693766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml (deflated 43%)
2022-11-23T02:20:52.3694413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml (deflated 43%)
2022-11-23T02:20:52.3695184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013159.xml (deflated 41%)
2022-11-23T02:20:52.3695842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013203.xml (deflated 43%)
2022-11-23T02:20:52.3696501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013204.xml (deflated 43%)
2022-11-23T02:20:52.3697152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013206.xml (deflated 41%)
2022-11-23T02:20:52.3697789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013210.xml (deflated 42%)
2022-11-23T02:20:52.3698447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml (deflated 41%)
2022-11-23T02:20:52.3699100Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013218.xml (deflated 41%)
2022-11-23T02:20:52.3699751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013222.xml (deflated 42%)
2022-11-23T02:20:52.3700368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013223.xml (deflated 42%)
2022-11-23T02:20:52.3701013Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013225.xml (deflated 41%)
2022-11-23T02:20:52.3701731Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013229.xml (deflated 41%)
2022-11-23T02:20:52.3702379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013233.xml (deflated 41%)
2022-11-23T02:20:52.3703014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013237.xml (deflated 41%)
2022-11-23T02:20:52.3703652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013241.xml (deflated 41%)
2022-11-23T02:20:52.3704355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013245.xml (deflated 41%)
2022-11-23T02:20:52.3705002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml (deflated 41%)
2022-11-23T02:20:52.3705628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013253.xml (deflated 41%)
2022-11-23T02:20:52.3706259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013257.xml (deflated 42%)
2022-11-23T02:20:52.3706901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml (deflated 42%)
2022-11-23T02:20:52.3707554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013300.xml (deflated 42%)
2022-11-23T02:20:52.3708192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013301.xml (deflated 41%)
2022-11-23T02:20:52.3708840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013306.xml (deflated 41%)
2022-11-23T02:20:52.3709513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013309.xml (deflated 41%)
2022-11-23T02:20:52.3710167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013313.xml (deflated 41%)
2022-11-23T02:20:52.3710809Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013317.xml (deflated 41%)
2022-11-23T02:20:52.3711458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013321.xml (deflated 42%)
2022-11-23T02:20:52.3712120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml (deflated 42%)
2022-11-23T02:20:52.3712777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013329.xml (deflated 41%)
2022-11-23T02:20:52.3713414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013333.xml (deflated 42%)
2022-11-23T02:20:52.3714060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013337.xml (deflated 43%)
2022-11-23T02:20:52.3714702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013341.xml (deflated 41%)
2022-11-23T02:20:52.3715359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml (deflated 41%)
2022-11-23T02:20:52.3715987Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123013408.xml (deflated 77%)
2022-11-23T02:20:52.3716687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013414.xml (deflated 42%)
2022-11-23T02:20:52.3717423Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013416.xml (deflated 42%)
2022-11-23T02:20:52.3718170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20221123013417.xml (deflated 42%)
2022-11-23T02:20:52.3718868Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013419.xml (deflated 43%)
2022-11-23T02:20:52.3719562Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013421.xml (deflated 43%)
2022-11-23T02:20:52.3720257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013423.xml (deflated 44%)
2022-11-23T02:20:52.3721001Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013424.xml (deflated 44%)
2022-11-23T02:20:52.3721725Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013426.xml (deflated 43%)
2022-11-23T02:20:52.3722406Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123013428.xml (deflated 44%)
2022-11-23T02:20:52.3723091Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20221123013430.xml (deflated 61%)
2022-11-23T02:20:52.3723810Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20221123013431.xml (deflated 62%)
2022-11-23T02:20:52.3724555Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221123013432.xml (deflated 75%)
2022-11-23T02:20:52.3725423Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123013434.xml (deflated 44%)
2022-11-23T02:20:52.3726173Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123013435.xml (deflated 62%)
2022-11-23T02:20:52.3726893Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013438.xml (deflated 42%)
2022-11-23T02:20:52.3727590Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013440.xml (deflated 42%)
2022-11-23T02:20:52.3728278Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013442.xml (deflated 42%)
2022-11-23T02:20:52.3728996Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221123013444.xml (deflated 42%)
2022-11-23T02:20:52.3729706Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013445.xml (deflated 43%)
2022-11-23T02:20:52.3730414Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013447.xml (deflated 43%)
2022-11-23T02:20:52.3731118Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013449.xml (deflated 43%)
2022-11-23T02:20:52.3731815Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013451.xml (deflated 43%)
2022-11-23T02:20:52.3732487Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013452.xml (deflated 43%)
2022-11-23T02:20:52.3733260Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013454.xml (deflated 43%)
2022-11-23T02:20:52.3734045Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013456.xml (deflated 43%)
2022-11-23T02:20:52.3734742Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013458.xml (deflated 43%)
2022-11-23T02:20:52.3735549Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123013459.xml (deflated 42%)
2022-11-23T02:20:52.3736356Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123013501.xml (deflated 82%)
2022-11-23T02:20:52.3737147Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20221123013503.xml (deflated 69%)
2022-11-23T02:20:52.3737814Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20221123013503.xml (deflated 62%)
2022-11-23T02:20:52.3738532Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123013504.xml (deflated 71%)
2022-11-23T02:20:52.3739214Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123013506.xml (deflated 90%)
2022-11-23T02:20:52.3739903Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20221123013506.xml (deflated 91%)
2022-11-23T02:20:52.3740566Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20221123013508.xml (deflated 43%)
2022-11-23T02:20:52.3741200Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123013509.xml (deflated 63%)
2022-11-23T02:20:52.3741826Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123013511.xml (deflated 55%)
2022-11-23T02:20:52.3742491Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20221123013513.xml (deflated 43%)
2022-11-23T02:20:52.3743105Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221123013513.xml (deflated 69%)
2022-11-23T02:20:52.3743762Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20221123013515.xml (deflated 72%)
2022-11-23T02:20:52.3744464Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20221123013515.xml (deflated 41%)
2022-11-23T02:20:52.3745134Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123013516.xml (deflated 43%)
2022-11-23T02:20:52.3745840Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123013520.xml (deflated 67%)
2022-11-23T02:20:52.3746524Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123013520.xml (deflated 79%)
2022-11-23T02:20:52.3747202Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123013524.xml (deflated 46%)
2022-11-23T02:20:52.3747894Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221123013527.xml (deflated 42%)
2022-11-23T02:20:52.3748520Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221123013532.xml (deflated 41%)
2022-11-23T02:20:52.3749176Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221123013536.xml (deflated 71%)
2022-11-23T02:20:52.3749821Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221123013541.xml (deflated 59%)
2022-11-23T02:20:52.3750561Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221123013546.xml (deflated 62%)
2022-11-23T02:20:52.3751280Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123013551.xml (deflated 66%)
2022-11-23T02:20:52.3751949Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123013551.xml (deflated 69%)
2022-11-23T02:20:52.3752677Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123013551.xml (deflated 66%)
2022-11-23T02:20:52.3753483Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123013556.xml (deflated 44%)
2022-11-23T02:20:52.3754233Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123013556.xml (deflated 44%)
2022-11-23T02:20:52.3754936Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123013556.xml (deflated 61%)
2022-11-23T02:20:52.3755626Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123013556.xml (deflated 60%)
2022-11-23T02:20:52.3756312Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123013556.xml (deflated 60%)
2022-11-23T02:20:52.3757022Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123013556.xml (deflated 62%)
2022-11-23T02:20:52.3757706Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123013556.xml (deflated 62%)
2022-11-23T02:20:52.3758405Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123013556.xml (deflated 90%)
2022-11-23T02:20:52.3759144Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123013556.xml (deflated 70%)
2022-11-23T02:20:52.3759921Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123013556.xml (deflated 88%)
2022-11-23T02:20:52.3760686Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123013556.xml (deflated 86%)
2022-11-23T02:20:52.3761502Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123013556.xml (deflated 63%)
2022-11-23T02:20:52.3762186Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20221123013602.xml (deflated 64%)
2022-11-23T02:20:52.3762875Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20221123013608.xml (deflated 66%)
2022-11-23T02:20:52.3763589Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123013616.xml (deflated 69%)
2022-11-23T02:20:52.3764263Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123013624.xml (deflated 71%)
2022-11-23T02:20:52.3765001Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123013624.xml (deflated 40%)
2022-11-23T02:20:52.3765682Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123013633.xml (deflated 77%)
2022-11-23T02:20:52.3766644Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123013644.xml (deflated 80%)
2022-11-23T02:20:52.3767342Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123013655.xml (deflated 54%)
2022-11-23T02:20:52.3767962Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221123013708.xml (deflated 38%)
2022-11-23T02:20:52.3768597Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013711.xml (deflated 42%)
2022-11-23T02:20:52.3769290Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013713.xml (deflated 41%)
2022-11-23T02:20:52.3770035Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013715.xml (deflated 41%)
2022-11-23T02:20:52.3770692Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221123013716.xml (deflated 42%)
2022-11-23T02:20:52.3771399Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013718.xml (deflated 41%)
2022-11-23T02:20:52.3772107Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013722.xml (deflated 41%)
2022-11-23T02:20:52.3772822Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013726.xml (deflated 41%)
2022-11-23T02:20:52.3773540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221123013730.xml (deflated 41%)
2022-11-23T02:20:52.3774230Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013733.xml (deflated 39%)
2022-11-23T02:20:52.3774952Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013735.xml (deflated 39%)
2022-11-23T02:20:52.3775513Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221123013737.xml (deflated 39%)
2022-11-23T02:20:52.3776232Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_flatten_params/TEST-TestFlattenParams-20221123013739.xml (deflated 78%)
2022-11-23T02:20:52.3776922Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123013754.xml (deflated 80%)
2022-11-23T02:20:52.3777597Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221123013810.xml (deflated 91%)
2022-11-23T02:20:52.3778355Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123013827.xml (deflated 84%)
2022-11-23T02:20:52.3779093Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20221123013843.xml (deflated 42%)
2022-11-23T02:20:52.3779816Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20221123013843.xml (deflated 76%)
2022-11-23T02:20:52.3780606Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013902.xml (deflated 41%)
2022-11-23T02:20:52.3781357Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013905.xml (deflated 41%)
2022-11-23T02:20:52.3782136Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013909.xml (deflated 41%)
2022-11-23T02:20:52.3782885Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013913.xml (deflated 41%)
2022-11-23T02:20:52.3783624Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013916.xml (deflated 41%)
2022-11-23T02:20:52.3784491Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013920.xml (deflated 41%)
2022-11-23T02:20:52.3785225Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013924.xml (deflated 41%)
2022-11-23T02:20:52.3786021Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013927.xml (deflated 41%)
2022-11-23T02:20:52.3786772Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221123013931.xml (deflated 41%)
2022-11-23T02:20:52.3787652Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013935.xml (deflated 41%)
2022-11-23T02:20:52.3788393Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013936.xml (deflated 41%)
2022-11-23T02:20:52.3789113Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013938.xml (deflated 41%)
2022-11-23T02:20:52.3789897Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013939.xml (deflated 41%)
2022-11-23T02:20:52.3790656Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221123013940.xml (deflated 41%)
2022-11-23T02:20:52.3791364Z   adding: test/test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20221123013942.xml (deflated 80%)
2022-11-23T02:20:52.3792034Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123014005.xml (deflated 94%)
2022-11-23T02:20:52.3792708Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123014028.xml (deflated 81%)
2022-11-23T02:20:52.3793426Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123014056.xml (deflated 70%)
2022-11-23T02:20:52.3794097Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123014056.xml (deflated 76%)
2022-11-23T02:20:52.3794767Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20221123014125.xml (deflated 84%)
2022-11-23T02:20:52.3795444Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123014204.xml (deflated 87%)
2022-11-23T02:20:52.3796116Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123014204.xml (deflated 91%)
2022-11-23T02:20:52.3796761Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014248.xml (deflated 40%)
2022-11-23T02:20:52.3797375Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014249.xml (deflated 40%)
2022-11-23T02:20:52.3797995Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014250.xml (deflated 40%)
2022-11-23T02:20:52.3798620Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014252.xml (deflated 40%)
2022-11-23T02:20:52.3799248Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014253.xml (deflated 39%)
2022-11-23T02:20:52.3799852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014255.xml (deflated 40%)
2022-11-23T02:20:52.3800450Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014256.xml (deflated 39%)
2022-11-23T02:20:52.3801027Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014258.xml (deflated 40%)
2022-11-23T02:20:52.3801642Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014259.xml (deflated 39%)
2022-11-23T02:20:52.3802306Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014301.xml (deflated 39%)
2022-11-23T02:20:52.3802927Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014302.xml (deflated 39%)
2022-11-23T02:20:52.3803551Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014303.xml (deflated 40%)
2022-11-23T02:20:52.3804131Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014305.xml (deflated 40%)
2022-11-23T02:20:52.3804715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014306.xml (deflated 39%)
2022-11-23T02:20:52.3805430Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014308.xml (deflated 40%)
2022-11-23T02:20:52.3806046Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014309.xml (deflated 39%)
2022-11-23T02:20:52.3806656Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014311.xml (deflated 40%)
2022-11-23T02:20:52.3807285Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014312.xml (deflated 40%)
2022-11-23T02:20:52.3807913Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20221123014314.xml (deflated 40%)
2022-11-23T02:20:52.3808540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014315.xml (deflated 39%)
2022-11-23T02:20:52.3809149Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014318.xml (deflated 40%)
2022-11-23T02:20:52.3809788Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014322.xml (deflated 40%)
2022-11-23T02:20:52.3810425Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014325.xml (deflated 40%)
2022-11-23T02:20:52.3811064Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014328.xml (deflated 40%)
2022-11-23T02:20:52.3811689Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014332.xml (deflated 40%)
2022-11-23T02:20:52.3812288Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20221123014335.xml (deflated 40%)
2022-11-23T02:20:52.3812999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014339.xml (deflated 42%)
2022-11-23T02:20:52.3813766Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014340.xml (deflated 42%)
2022-11-23T02:20:52.3814493Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014342.xml (deflated 42%)
2022-11-23T02:20:52.3815325Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014343.xml (deflated 42%)
2022-11-23T02:20:52.3815999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014344.xml (deflated 42%)
2022-11-23T02:20:52.3816698Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014346.xml (deflated 42%)
2022-11-23T02:20:52.3817373Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014347.xml (deflated 42%)
2022-11-23T02:20:52.3818629Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014349.xml (deflated 42%)
2022-11-23T02:20:52.3819350Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014350.xml (deflated 43%)
2022-11-23T02:20:52.3820254Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014352.xml (deflated 44%)
2022-11-23T02:20:52.3820895Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014355.xml (deflated 44%)
2022-11-23T02:20:52.3821584Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014359.xml (deflated 43%)
2022-11-23T02:20:52.3822234Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014402.xml (deflated 43%)
2022-11-23T02:20:52.3822897Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014405.xml (deflated 45%)
2022-11-23T02:20:52.3823584Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014409.xml (deflated 45%)
2022-11-23T02:20:52.3824255Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014412.xml (deflated 47%)
2022-11-23T02:20:52.3824891Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014415.xml (deflated 46%)
2022-11-23T02:20:52.3825553Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014419.xml (deflated 45%)
2022-11-23T02:20:52.3826193Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014422.xml (deflated 45%)
2022-11-23T02:20:52.3826829Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014426.xml (deflated 45%)
2022-11-23T02:20:52.3827468Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014429.xml (deflated 44%)
2022-11-23T02:20:52.3828065Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014432.xml (deflated 44%)
2022-11-23T02:20:52.3828673Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014436.xml (deflated 42%)
2022-11-23T02:20:52.3829258Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014437.xml (deflated 42%)
2022-11-23T02:20:52.3829860Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014439.xml (deflated 42%)
2022-11-23T02:20:52.3830457Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014440.xml (deflated 44%)
2022-11-23T02:20:52.3831057Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014442.xml (deflated 44%)
2022-11-23T02:20:52.3831650Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014443.xml (deflated 42%)
2022-11-23T02:20:52.3832251Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014444.xml (deflated 44%)
2022-11-23T02:20:52.3832845Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014446.xml (deflated 42%)
2022-11-23T02:20:52.3833445Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014447.xml (deflated 42%)
2022-11-23T02:20:52.3834157Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014449.xml (deflated 42%)
2022-11-23T02:20:52.3835078Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014450.xml (deflated 42%)
2022-11-23T02:20:52.3836001Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014452.xml (deflated 43%)
2022-11-23T02:20:52.3836749Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014453.xml (deflated 42%)
2022-11-23T02:20:52.3837351Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014455.xml (deflated 41%)
2022-11-23T02:20:52.3837946Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014456.xml (deflated 41%)
2022-11-23T02:20:52.3838545Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014457.xml (deflated 41%)
2022-11-23T02:20:52.3839177Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014459.xml (deflated 41%)
2022-11-23T02:20:52.3839779Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014500.xml (deflated 42%)
2022-11-23T02:20:52.3840363Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014502.xml (deflated 42%)
2022-11-23T02:20:52.3840971Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014503.xml (deflated 42%)
2022-11-23T02:20:52.3859047Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014505.xml (deflated 42%)
2022-11-23T02:20:52.3859918Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014506.xml (deflated 42%)
2022-11-23T02:20:52.3860563Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014508.xml (deflated 42%)
2022-11-23T02:20:52.3861164Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014509.xml (deflated 42%)
2022-11-23T02:20:52.3861771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014510.xml (deflated 41%)
2022-11-23T02:20:52.3862386Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014514.xml (deflated 43%)
2022-11-23T02:20:52.3862999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014515.xml (deflated 43%)
2022-11-23T02:20:52.3863594Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014517.xml (deflated 42%)
2022-11-23T02:20:52.3864204Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014518.xml (deflated 43%)
2022-11-23T02:20:52.3864806Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014520.xml (deflated 43%)
2022-11-23T02:20:52.3865419Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014521.xml (deflated 43%)
2022-11-23T02:20:52.3866011Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014522.xml (deflated 43%)
2022-11-23T02:20:52.3866605Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014524.xml (deflated 43%)
2022-11-23T02:20:52.3867210Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014525.xml (deflated 43%)
2022-11-23T02:20:52.3867808Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014527.xml (deflated 43%)
2022-11-23T02:20:52.3868402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014528.xml (deflated 43%)
2022-11-23T02:20:52.3869105Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014530.xml (deflated 44%)
2022-11-23T02:20:52.3869710Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014531.xml (deflated 42%)
2022-11-23T02:20:52.3870310Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014533.xml (deflated 43%)
2022-11-23T02:20:52.3870897Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014534.xml (deflated 41%)
2022-11-23T02:20:52.3871500Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014535.xml (deflated 41%)
2022-11-23T02:20:52.3872138Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014537.xml (deflated 42%)
2022-11-23T02:20:52.3872739Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20221123014540.xml (deflated 42%)
2022-11-23T02:20:52.3873317Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014544.xml (deflated 41%)
2022-11-23T02:20:52.3873891Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014545.xml (deflated 41%)
2022-11-23T02:20:52.3874458Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014546.xml (deflated 41%)
2022-11-23T02:20:52.3875022Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014548.xml (deflated 42%)
2022-11-23T02:20:52.3875569Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014549.xml (deflated 41%)
2022-11-23T02:20:52.3876136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014551.xml (deflated 41%)
2022-11-23T02:20:52.3876706Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014552.xml (deflated 42%)
2022-11-23T02:20:52.3877266Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014554.xml (deflated 42%)
2022-11-23T02:20:52.3877814Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20221123014555.xml (deflated 42%)
2022-11-23T02:20:52.3878481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014557.xml (deflated 43%)
2022-11-23T02:20:52.3879240Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014558.xml (deflated 43%)
2022-11-23T02:20:52.3879992Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014559.xml (deflated 44%)
2022-11-23T02:20:52.3880739Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20221123014601.xml (deflated 43%)
2022-11-23T02:20:52.3881389Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20221123014602.xml (deflated 42%)
2022-11-23T02:20:52.3881979Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014604.xml (deflated 41%)
2022-11-23T02:20:52.3882549Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014605.xml (deflated 41%)
2022-11-23T02:20:52.3883110Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014607.xml (deflated 41%)
2022-11-23T02:20:52.3883660Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014608.xml (deflated 42%)
2022-11-23T02:20:52.3884257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014610.xml (deflated 42%)
2022-11-23T02:20:52.3884829Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014611.xml (deflated 42%)
2022-11-23T02:20:52.3885471Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014612.xml (deflated 42%)
2022-11-23T02:20:52.3886014Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014614.xml (deflated 41%)
2022-11-23T02:20:52.3886609Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014615.xml (deflated 41%)
2022-11-23T02:20:52.3887170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014617.xml (deflated 41%)
2022-11-23T02:20:52.3887724Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014618.xml (deflated 42%)
2022-11-23T02:20:52.3888269Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014620.xml (deflated 41%)
2022-11-23T02:20:52.3888892Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014621.xml (deflated 41%)
2022-11-23T02:20:52.3889453Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014623.xml (deflated 41%)
2022-11-23T02:20:52.3890006Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014624.xml (deflated 41%)
2022-11-23T02:20:52.3890557Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014625.xml (deflated 41%)
2022-11-23T02:20:52.3891113Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014627.xml (deflated 41%)
2022-11-23T02:20:52.3891672Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014628.xml (deflated 41%)
2022-11-23T02:20:52.3892224Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20221123014630.xml (deflated 42%)
2022-11-23T02:20:52.3892766Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20221123014631.xml (deflated 41%)
2022-11-23T02:20:52.3893294Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20221123014633.xml (deflated 40%)
2022-11-23T02:20:52.3893854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123014634.xml (deflated 86%)
2022-11-23T02:20:52.3894435Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123014634.xml (deflated 86%)
2022-11-23T02:20:52.3895179Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014735.xml (deflated 42%)
2022-11-23T02:20:52.3895838Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014737.xml (deflated 42%)
2022-11-23T02:20:52.3896495Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014739.xml (deflated 41%)
2022-11-23T02:20:52.3897143Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20221123014744.xml (deflated 41%)
2022-11-23T02:20:52.3897814Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014748.xml (deflated 42%)
2022-11-23T02:20:52.3898516Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014754.xml (deflated 42%)
2022-11-23T02:20:52.3899264Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014800.xml (deflated 42%)
2022-11-23T02:20:52.3899957Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20221123014805.xml (deflated 41%)
2022-11-23T02:20:52.3900604Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014811.xml (deflated 43%)
2022-11-23T02:20:52.3901238Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014816.xml (deflated 41%)
2022-11-23T02:20:52.3901912Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014820.xml (deflated 41%)
2022-11-23T02:20:52.3902555Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014825.xml (deflated 41%)
2022-11-23T02:20:52.3903178Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014829.xml (deflated 40%)
2022-11-23T02:20:52.3903809Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014834.xml (deflated 40%)
2022-11-23T02:20:52.3904446Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014838.xml (deflated 41%)
2022-11-23T02:20:52.3905087Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014842.xml (deflated 40%)
2022-11-23T02:20:52.3905721Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014847.xml (deflated 42%)
2022-11-23T02:20:52.3906350Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014849.xml (deflated 40%)
2022-11-23T02:20:52.3906987Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014853.xml (deflated 41%)
2022-11-23T02:20:52.3907624Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014855.xml (deflated 41%)
2022-11-23T02:20:52.3908257Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014900.xml (deflated 41%)
2022-11-23T02:20:52.3908878Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014905.xml (deflated 41%)
2022-11-23T02:20:52.3909514Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014909.xml (deflated 40%)
2022-11-23T02:20:52.3910147Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014914.xml (deflated 40%)
2022-11-23T02:20:52.3910776Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014918.xml (deflated 40%)
2022-11-23T02:20:52.3911391Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014922.xml (deflated 41%)
2022-11-23T02:20:52.3912025Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014927.xml (deflated 41%)
2022-11-23T02:20:52.3912660Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014931.xml (deflated 40%)
2022-11-23T02:20:52.3913292Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014936.xml (deflated 40%)
2022-11-23T02:20:52.3913941Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014940.xml (deflated 40%)
2022-11-23T02:20:52.3914574Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014945.xml (deflated 40%)
2022-11-23T02:20:52.3915208Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123014959.xml (deflated 42%)
2022-11-23T02:20:52.3915833Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015001.xml (deflated 40%)
2022-11-23T02:20:52.3916497Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015006.xml (deflated 40%)
2022-11-23T02:20:52.3917124Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015010.xml (deflated 40%)
2022-11-23T02:20:52.3917759Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015015.xml (deflated 40%)
2022-11-23T02:20:52.3918395Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015019.xml (deflated 49%)
2022-11-23T02:20:52.3919026Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015024.xml (deflated 41%)
2022-11-23T02:20:52.3919643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015028.xml (deflated 42%)
2022-11-23T02:20:52.3920276Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015033.xml (deflated 42%)
2022-11-23T02:20:52.3920911Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015038.xml (deflated 41%)
2022-11-23T02:20:52.3921541Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015042.xml (deflated 41%)
2022-11-23T02:20:52.3922155Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015047.xml (deflated 43%)
2022-11-23T02:20:52.3922789Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015049.xml (deflated 41%)
2022-11-23T02:20:52.3923425Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015053.xml (deflated 41%)
2022-11-23T02:20:52.3924058Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015058.xml (deflated 41%)
2022-11-23T02:20:52.3924677Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015102.xml (deflated 41%)
2022-11-23T02:20:52.3925383Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015107.xml (deflated 41%)
2022-11-23T02:20:52.3926014Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015112.xml (deflated 41%)
2022-11-23T02:20:52.3926643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015116.xml (deflated 41%)
2022-11-23T02:20:52.3927261Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015121.xml (deflated 41%)
2022-11-23T02:20:52.3927889Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015126.xml (deflated 41%)
2022-11-23T02:20:52.3928556Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015130.xml (deflated 41%)
2022-11-23T02:20:52.3929189Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015135.xml (deflated 41%)
2022-11-23T02:20:52.3929821Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015139.xml (deflated 40%)
2022-11-23T02:20:52.3930437Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015144.xml (deflated 41%)
2022-11-23T02:20:52.3931097Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015148.xml (deflated 41%)
2022-11-23T02:20:52.3931721Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015154.xml (deflated 41%)
2022-11-23T02:20:52.3932354Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015158.xml (deflated 41%)
2022-11-23T02:20:52.3933124Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015203.xml (deflated 41%)
2022-11-23T02:20:52.3933985Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015207.xml (deflated 44%)
2022-11-23T02:20:52.3934620Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015211.xml (deflated 41%)
2022-11-23T02:20:52.3935387Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015216.xml (deflated 41%)
2022-11-23T02:20:52.3936016Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015220.xml (deflated 42%)
2022-11-23T02:20:52.3936646Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015225.xml (deflated 41%)
2022-11-23T02:20:52.3937281Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015230.xml (deflated 41%)
2022-11-23T02:20:52.3937914Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015234.xml (deflated 41%)
2022-11-23T02:20:52.3938535Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015238.xml (deflated 41%)
2022-11-23T02:20:52.3939164Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015244.xml (deflated 41%)
2022-11-23T02:20:52.3939805Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015249.xml (deflated 41%)
2022-11-23T02:20:52.3940438Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20221123015253.xml (deflated 41%)
2022-11-23T02:20:52.3941068Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015258.xml (deflated 43%)
2022-11-23T02:20:52.3941717Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015300.xml (deflated 42%)
2022-11-23T02:20:52.3942363Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015304.xml (deflated 41%)
2022-11-23T02:20:52.3943003Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20221123015309.xml (deflated 42%)
2022-11-23T02:20:52.3943737Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015311.xml (deflated 41%)
2022-11-23T02:20:52.3944387Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015315.xml (deflated 42%)
2022-11-23T02:20:52.3945049Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015320.xml (deflated 41%)
2022-11-23T02:20:52.3945712Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20221123015324.xml (deflated 41%)
2022-11-23T02:20:52.3946379Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015329.xml (deflated 40%)
2022-11-23T02:20:52.3946972Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015333.xml (deflated 39%)
2022-11-23T02:20:52.3947575Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015337.xml (deflated 40%)
2022-11-23T02:20:52.3948181Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015342.xml (deflated 40%)
2022-11-23T02:20:52.3948778Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015346.xml (deflated 40%)
2022-11-23T02:20:52.3949367Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015351.xml (deflated 40%)
2022-11-23T02:20:52.3949966Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015356.xml (deflated 40%)
2022-11-23T02:20:52.3950569Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015400.xml (deflated 40%)
2022-11-23T02:20:52.3951176Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015404.xml (deflated 40%)
2022-11-23T02:20:52.3951765Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015409.xml (deflated 40%)
2022-11-23T02:20:52.3952359Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015413.xml (deflated 40%)
2022-11-23T02:20:52.3952952Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015418.xml (deflated 40%)
2022-11-23T02:20:52.3953550Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015422.xml (deflated 40%)
2022-11-23T02:20:52.3954131Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015426.xml (deflated 40%)
2022-11-23T02:20:52.3954733Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015431.xml (deflated 40%)
2022-11-23T02:20:52.3955330Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015435.xml (deflated 40%)
2022-11-23T02:20:52.3955931Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015440.xml (deflated 40%)
2022-11-23T02:20:52.3956526Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015444.xml (deflated 40%)
2022-11-23T02:20:52.3957114Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015449.xml (deflated 40%)
2022-11-23T02:20:52.3957713Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015453.xml (deflated 39%)
2022-11-23T02:20:52.3958359Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015458.xml (deflated 41%)
2022-11-23T02:20:52.3958952Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015502.xml (deflated 40%)
2022-11-23T02:20:52.3959533Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015507.xml (deflated 40%)
2022-11-23T02:20:52.3960129Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015511.xml (deflated 41%)
2022-11-23T02:20:52.3960753Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015513.xml (deflated 40%)
2022-11-23T02:20:52.3961349Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015518.xml (deflated 41%)
2022-11-23T02:20:52.3961939Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015522.xml (deflated 40%)
2022-11-23T02:20:52.3962538Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015526.xml (deflated 39%)
2022-11-23T02:20:52.3963133Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015531.xml (deflated 40%)
2022-11-23T02:20:52.3963728Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015535.xml (deflated 40%)
2022-11-23T02:20:52.3964313Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015540.xml (deflated 40%)
2022-11-23T02:20:52.3964981Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015544.xml (deflated 39%)
2022-11-23T02:20:52.3965588Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015549.xml (deflated 39%)
2022-11-23T02:20:52.3966184Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015554.xml (deflated 40%)
2022-11-23T02:20:52.3966771Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015558.xml (deflated 40%)
2022-11-23T02:20:52.3967360Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015603.xml (deflated 40%)
2022-11-23T02:20:52.3967961Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015608.xml (deflated 40%)
2022-11-23T02:20:52.3968556Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015612.xml (deflated 41%)
2022-11-23T02:20:52.3969139Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015617.xml (deflated 40%)
2022-11-23T02:20:52.3969732Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015621.xml (deflated 40%)
2022-11-23T02:20:52.3970326Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015625.xml (deflated 40%)
2022-11-23T02:20:52.3970919Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015630.xml (deflated 40%)
2022-11-23T02:20:52.3971504Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015634.xml (deflated 40%)
2022-11-23T02:20:52.3972098Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015639.xml (deflated 40%)
2022-11-23T02:20:52.3972723Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015643.xml (deflated 40%)
2022-11-23T02:20:52.3973322Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015648.xml (deflated 40%)
2022-11-23T02:20:52.3973913Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015652.xml (deflated 39%)
2022-11-23T02:20:52.3974497Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015657.xml (deflated 39%)
2022-11-23T02:20:52.3975244Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015701.xml (deflated 40%)
2022-11-23T02:20:52.3975845Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015706.xml (deflated 40%)
2022-11-23T02:20:52.3976439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015710.xml (deflated 40%)
2022-11-23T02:20:52.3977027Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015715.xml (deflated 40%)
2022-11-23T02:20:52.3977625Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015719.xml (deflated 40%)
2022-11-23T02:20:52.3978225Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20221123015724.xml (deflated 40%)
2022-11-23T02:20:52.3978859Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20221123015728.xml (deflated 43%)
2022-11-23T02:20:52.3979548Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20221123015733.xml (deflated 43%)
2022-11-23T02:20:52.3980230Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015737.xml (deflated 41%)
2022-11-23T02:20:52.3980870Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015742.xml (deflated 41%)
2022-11-23T02:20:52.3981508Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015746.xml (deflated 40%)
2022-11-23T02:20:52.3982126Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015750.xml (deflated 41%)
2022-11-23T02:20:52.3982758Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015754.xml (deflated 40%)
2022-11-23T02:20:52.3983399Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015758.xml (deflated 41%)
2022-11-23T02:20:52.3984034Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015803.xml (deflated 41%)
2022-11-23T02:20:52.3984654Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015807.xml (deflated 41%)
2022-11-23T02:20:52.3985290Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015811.xml (deflated 41%)
2022-11-23T02:20:52.3985925Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015815.xml (deflated 41%)
2022-11-23T02:20:52.3986564Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015819.xml (deflated 42%)
2022-11-23T02:20:52.3987238Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015823.xml (deflated 41%)
2022-11-23T02:20:52.3987865Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20221123015827.xml (deflated 41%)
2022-11-23T02:20:52.3988478Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015832.xml (deflated 40%)
2022-11-23T02:20:52.3989069Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015836.xml (deflated 40%)
2022-11-23T02:20:52.3989687Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015840.xml (deflated 40%)
2022-11-23T02:20:52.3990255Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015845.xml (deflated 40%)
2022-11-23T02:20:52.3990845Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015849.xml (deflated 40%)
2022-11-23T02:20:52.3991429Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015903.xml (deflated 40%)
2022-11-23T02:20:52.3992009Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015907.xml (deflated 40%)
2022-11-23T02:20:52.3992574Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015912.xml (deflated 40%)
2022-11-23T02:20:52.3993158Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015916.xml (deflated 40%)
2022-11-23T02:20:52.3993737Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015921.xml (deflated 40%)
2022-11-23T02:20:52.3994317Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015925.xml (deflated 40%)
2022-11-23T02:20:52.3994879Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015929.xml (deflated 40%)
2022-11-23T02:20:52.3995461Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015934.xml (deflated 39%)
2022-11-23T02:20:52.3996040Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015938.xml (deflated 39%)
2022-11-23T02:20:52.3996612Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015943.xml (deflated 40%)
2022-11-23T02:20:52.3997181Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015947.xml (deflated 39%)
2022-11-23T02:20:52.3997758Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015951.xml (deflated 39%)
2022-11-23T02:20:52.3998340Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123015956.xml (deflated 40%)
2022-11-23T02:20:52.3998914Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020000.xml (deflated 40%)
2022-11-23T02:20:52.3999478Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020005.xml (deflated 40%)
2022-11-23T02:20:52.4000047Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020009.xml (deflated 40%)
2022-11-23T02:20:52.4000630Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020014.xml (deflated 40%)
2022-11-23T02:20:52.4001228Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020018.xml (deflated 41%)
2022-11-23T02:20:52.4001792Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020020.xml (deflated 40%)
2022-11-23T02:20:52.4002369Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020025.xml (deflated 40%)
2022-11-23T02:20:52.4002949Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020029.xml (deflated 40%)
2022-11-23T02:20:52.4003524Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020034.xml (deflated 40%)
2022-11-23T02:20:52.4004112Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020038.xml (deflated 40%)
2022-11-23T02:20:52.4004683Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020043.xml (deflated 40%)
2022-11-23T02:20:52.4005330Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020047.xml (deflated 40%)
2022-11-23T02:20:52.4005900Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020051.xml (deflated 40%)
2022-11-23T02:20:52.4006472Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020056.xml (deflated 39%)
2022-11-23T02:20:52.4006790Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020101.xml (deflated 40%)
2022-11-23T02:20:52.4007122Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020107.xml (deflated 40%)
2022-11-23T02:20:52.4007453Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020111.xml (deflated 40%)
2022-11-23T02:20:52.4007784Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020116.xml (deflated 40%)
2022-11-23T02:20:52.4008112Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020120.xml (deflated 41%)
2022-11-23T02:20:52.4008439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020124.xml (deflated 40%)
2022-11-23T02:20:52.4008770Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020128.xml (deflated 42%)
2022-11-23T02:20:52.4009097Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020133.xml (deflated 40%)
2022-11-23T02:20:52.4009421Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020137.xml (deflated 40%)
2022-11-23T02:20:52.4009749Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020142.xml (deflated 40%)
2022-11-23T02:20:52.4010062Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020146.xml (deflated 40%)
2022-11-23T02:20:52.4010390Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020151.xml (deflated 40%)
2022-11-23T02:20:52.4010719Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020155.xml (deflated 40%)
2022-11-23T02:20:52.4011048Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020159.xml (deflated 40%)
2022-11-23T02:20:52.4011376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020204.xml (deflated 40%)
2022-11-23T02:20:52.4011730Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020208.xml (deflated 40%)
2022-11-23T02:20:52.4012062Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020212.xml (deflated 40%)
2022-11-23T02:20:52.4012392Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020216.xml (deflated 43%)
2022-11-23T02:20:52.4012717Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020221.xml (deflated 41%)
2022-11-23T02:20:52.4013079Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020226.xml (deflated 40%)
2022-11-23T02:20:52.4013401Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020231.xml (deflated 45%)
2022-11-23T02:20:52.4013716Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020237.xml (deflated 40%)
2022-11-23T02:20:52.4014045Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020241.xml (deflated 41%)
2022-11-23T02:20:52.4014376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020246.xml (deflated 40%)
2022-11-23T02:20:52.4014705Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020250.xml (deflated 41%)
2022-11-23T02:20:52.4015155Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020254.xml (deflated 40%)
2022-11-23T02:20:52.4015490Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020259.xml (deflated 40%)
2022-11-23T02:20:52.4015822Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020303.xml (deflated 40%)
2022-11-23T02:20:52.4016150Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020307.xml (deflated 40%)
2022-11-23T02:20:52.4016478Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020312.xml (deflated 40%)
2022-11-23T02:20:52.4016807Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020316.xml (deflated 40%)
2022-11-23T02:20:52.4017141Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020320.xml (deflated 41%)
2022-11-23T02:20:52.4017457Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020322.xml (deflated 40%)
2022-11-23T02:20:52.4017789Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020327.xml (deflated 41%)
2022-11-23T02:20:52.4018115Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020331.xml (deflated 40%)
2022-11-23T02:20:52.4018439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020336.xml (deflated 40%)
2022-11-23T02:20:52.4018763Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020340.xml (deflated 40%)
2022-11-23T02:20:52.4019090Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020345.xml (deflated 40%)
2022-11-23T02:20:52.4019416Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020349.xml (deflated 40%)
2022-11-23T02:20:52.4019790Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020354.xml (deflated 40%)
2022-11-23T02:20:52.4020121Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020358.xml (deflated 41%)
2022-11-23T02:20:52.4020449Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020402.xml (deflated 40%)
2022-11-23T02:20:52.4020764Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020407.xml (deflated 40%)
2022-11-23T02:20:52.4021090Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020411.xml (deflated 40%)
2022-11-23T02:20:52.4021451Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020416.xml (deflated 40%)
2022-11-23T02:20:52.4021777Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020420.xml (deflated 40%)
2022-11-23T02:20:52.4022106Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020424.xml (deflated 40%)
2022-11-23T02:20:52.4022436Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020429.xml (deflated 40%)
2022-11-23T02:20:52.4022762Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020433.xml (deflated 40%)
2022-11-23T02:20:52.4023090Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020438.xml (deflated 40%)
2022-11-23T02:20:52.4023421Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020442.xml (deflated 40%)
2022-11-23T02:20:52.4023749Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020447.xml (deflated 40%)
2022-11-23T02:20:52.4024073Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020451.xml (deflated 40%)
2022-11-23T02:20:52.4024391Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020456.xml (deflated 40%)
2022-11-23T02:20:52.4024719Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020500.xml (deflated 40%)
2022-11-23T02:20:52.4025046Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020505.xml (deflated 41%)
2022-11-23T02:20:52.4025376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020509.xml (deflated 40%)
2022-11-23T02:20:52.4025709Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020513.xml (deflated 40%)
2022-11-23T02:20:52.4026038Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020518.xml (deflated 40%)
2022-11-23T02:20:52.4026368Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020522.xml (deflated 39%)
2022-11-23T02:20:52.4026696Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020527.xml (deflated 40%)
2022-11-23T02:20:52.4027025Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020531.xml (deflated 40%)
2022-11-23T02:20:52.4027357Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020535.xml (deflated 40%)
2022-11-23T02:20:52.4027672Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020540.xml (deflated 40%)
2022-11-23T02:20:52.4028023Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020544.xml (deflated 41%)
2022-11-23T02:20:52.4028354Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020549.xml (deflated 41%)
2022-11-23T02:20:52.4028679Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020553.xml (deflated 40%)
2022-11-23T02:20:52.4029007Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020557.xml (deflated 40%)
2022-11-23T02:20:52.4029362Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020602.xml (deflated 40%)
2022-11-23T02:20:52.4029689Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020606.xml (deflated 40%)
2022-11-23T02:20:52.4030015Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020610.xml (deflated 40%)
2022-11-23T02:20:52.4030341Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020615.xml (deflated 40%)
2022-11-23T02:20:52.4030669Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020622.xml (deflated 40%)
2022-11-23T02:20:52.4030992Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020630.xml (deflated 40%)
2022-11-23T02:20:52.4031307Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020634.xml (deflated 39%)
2022-11-23T02:20:52.4031632Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020638.xml (deflated 40%)
2022-11-23T02:20:52.4031959Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020643.xml (deflated 39%)
2022-11-23T02:20:52.4032289Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020647.xml (deflated 40%)
2022-11-23T02:20:52.4032616Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020654.xml (deflated 40%)
2022-11-23T02:20:52.4033071Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020702.xml (deflated 40%)
2022-11-23T02:20:52.4033386Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020706.xml (deflated 40%)
2022-11-23T02:20:52.4033711Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020711.xml (deflated 40%)
2022-11-23T02:20:52.4034042Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020715.xml (deflated 40%)
2022-11-23T02:20:52.4034365Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020720.xml (deflated 40%)
2022-11-23T02:20:52.4034687Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020724.xml (deflated 40%)
2022-11-23T02:20:52.4035015Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020728.xml (deflated 40%)
2022-11-23T02:20:52.4035337Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020733.xml (deflated 40%)
2022-11-23T02:20:52.4035668Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020737.xml (deflated 40%)
2022-11-23T02:20:52.4036029Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020743.xml (deflated 39%)
2022-11-23T02:20:52.4036356Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020750.xml (deflated 40%)
2022-11-23T02:20:52.4036681Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020754.xml (deflated 40%)
2022-11-23T02:20:52.4036995Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020759.xml (deflated 40%)
2022-11-23T02:20:52.4037344Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020803.xml (deflated 40%)
2022-11-23T02:20:52.4037669Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020807.xml (deflated 40%)
2022-11-23T02:20:52.4037998Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020812.xml (deflated 40%)
2022-11-23T02:20:52.4038324Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020816.xml (deflated 40%)
2022-11-23T02:20:52.4038650Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020820.xml (deflated 40%)
2022-11-23T02:20:52.4038977Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020825.xml (deflated 40%)
2022-11-23T02:20:52.4039306Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020829.xml (deflated 40%)
2022-11-23T02:20:52.4039637Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020834.xml (deflated 40%)
2022-11-23T02:20:52.4039964Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020838.xml (deflated 40%)
2022-11-23T02:20:52.4040280Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020842.xml (deflated 40%)
2022-11-23T02:20:52.4040606Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020847.xml (deflated 40%)
2022-11-23T02:20:52.4040932Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020851.xml (deflated 41%)
2022-11-23T02:20:52.4041256Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020856.xml (deflated 41%)
2022-11-23T02:20:52.4041585Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020900.xml (deflated 41%)
2022-11-23T02:20:52.4041914Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020905.xml (deflated 40%)
2022-11-23T02:20:52.4042240Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020909.xml (deflated 40%)
2022-11-23T02:20:52.4042567Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020913.xml (deflated 41%)
2022-11-23T02:20:52.4042898Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020917.xml (deflated 40%)
2022-11-23T02:20:52.4043221Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020922.xml (deflated 40%)
2022-11-23T02:20:52.4043551Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020926.xml (deflated 40%)
2022-11-23T02:20:52.4043886Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020931.xml (deflated 40%)
2022-11-23T02:20:52.4044213Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020935.xml (deflated 40%)
2022-11-23T02:20:52.4044529Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020940.xml (deflated 40%)
2022-11-23T02:20:52.4044859Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020944.xml (deflated 40%)
2022-11-23T02:20:52.4045274Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020949.xml (deflated 40%)
2022-11-23T02:20:52.4045636Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020953.xml (deflated 40%)
2022-11-23T02:20:52.4045965Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123020957.xml (deflated 40%)
2022-11-23T02:20:52.4046290Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021002.xml (deflated 40%)
2022-11-23T02:20:52.4046618Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021006.xml (deflated 40%)
2022-11-23T02:20:52.4046941Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021011.xml (deflated 40%)
2022-11-23T02:20:52.4047257Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021016.xml (deflated 40%)
2022-11-23T02:20:52.4047588Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021022.xml (deflated 41%)
2022-11-23T02:20:52.4047917Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021026.xml (deflated 41%)
2022-11-23T02:20:52.4048244Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021030.xml (deflated 41%)
2022-11-23T02:20:52.4048574Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021041.xml (deflated 40%)
2022-11-23T02:20:52.4048899Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021046.xml (deflated 40%)
2022-11-23T02:20:52.4049224Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021050.xml (deflated 40%)
2022-11-23T02:20:52.4049556Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021054.xml (deflated 40%)
2022-11-23T02:20:52.4049883Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021059.xml (deflated 40%)
2022-11-23T02:20:52.4050216Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021103.xml (deflated 40%)
2022-11-23T02:20:52.4050542Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021108.xml (deflated 40%)
2022-11-23T02:20:52.4050855Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021112.xml (deflated 40%)
2022-11-23T02:20:52.4051179Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021117.xml (deflated 41%)
2022-11-23T02:20:52.4051509Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021121.xml (deflated 41%)
2022-11-23T02:20:52.4051835Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021126.xml (deflated 40%)
2022-11-23T02:20:52.4052189Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021130.xml (deflated 41%)
2022-11-23T02:20:52.4052517Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021134.xml (deflated 40%)
2022-11-23T02:20:52.4052844Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021141.xml (deflated 40%)
2022-11-23T02:20:52.4053170Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021145.xml (deflated 40%)
2022-11-23T02:20:52.4053534Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021150.xml (deflated 40%)
2022-11-23T02:20:52.4053857Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021154.xml (deflated 39%)
2022-11-23T02:20:52.4054187Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021158.xml (deflated 40%)
2022-11-23T02:20:52.4054503Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021204.xml (deflated 40%)
2022-11-23T02:20:52.4054827Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021208.xml (deflated 40%)
2022-11-23T02:20:52.4055302Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021212.xml (deflated 40%)
2022-11-23T02:20:52.4055634Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021217.xml (deflated 40%)
2022-11-23T02:20:52.4055963Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021221.xml (deflated 40%)
2022-11-23T02:20:52.4056290Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021226.xml (deflated 41%)
2022-11-23T02:20:52.4056617Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021230.xml (deflated 40%)
2022-11-23T02:20:52.4056944Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021235.xml (deflated 42%)
2022-11-23T02:20:52.4057272Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021239.xml (deflated 41%)
2022-11-23T02:20:52.4057605Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021243.xml (deflated 40%)
2022-11-23T02:20:52.4057920Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021248.xml (deflated 41%)
2022-11-23T02:20:52.4058248Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021252.xml (deflated 40%)
2022-11-23T02:20:52.4058574Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021257.xml (deflated 40%)
2022-11-23T02:20:52.4058901Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021301.xml (deflated 39%)
2022-11-23T02:20:52.4059231Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021305.xml (deflated 40%)
2022-11-23T02:20:52.4059558Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021310.xml (deflated 40%)
2022-11-23T02:20:52.4059885Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021314.xml (deflated 40%)
2022-11-23T02:20:52.4060257Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021319.xml (deflated 41%)
2022-11-23T02:20:52.4060583Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021323.xml (deflated 40%)
2022-11-23T02:20:52.4060908Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021327.xml (deflated 40%)
2022-11-23T02:20:52.4061238Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021332.xml (deflated 40%)
2022-11-23T02:20:52.4061552Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021337.xml (deflated 40%)
2022-11-23T02:20:52.4061914Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021341.xml (deflated 40%)
2022-11-23T02:20:52.4062245Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021346.xml (deflated 40%)
2022-11-23T02:20:52.4062571Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021350.xml (deflated 40%)
2022-11-23T02:20:52.4062900Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021355.xml (deflated 40%)
2022-11-23T02:20:52.4063226Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021359.xml (deflated 40%)
2022-11-23T02:20:52.4063553Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021404.xml (deflated 40%)
2022-11-23T02:20:52.4063880Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021408.xml (deflated 40%)
2022-11-23T02:20:52.4064206Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021413.xml (deflated 40%)
2022-11-23T02:20:52.4064532Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021418.xml (deflated 40%)
2022-11-23T02:20:52.4064861Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021422.xml (deflated 40%)
2022-11-23T02:20:52.4065174Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021426.xml (deflated 40%)
2022-11-23T02:20:52.4065498Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021431.xml (deflated 40%)
2022-11-23T02:20:52.4065823Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021439.xml (deflated 39%)
2022-11-23T02:20:52.4066155Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021443.xml (deflated 40%)
2022-11-23T02:20:52.4066479Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021447.xml (deflated 40%)
2022-11-23T02:20:52.4066805Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021452.xml (deflated 40%)
2022-11-23T02:20:52.4067131Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20221123021456.xml (deflated 40%)
2022-11-23T02:20:52.4067560Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021500.xml (deflated 44%)
2022-11-23T02:20:52.4067979Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021505.xml (deflated 43%)
2022-11-23T02:20:52.4068445Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021509.xml (deflated 44%)
2022-11-23T02:20:52.4068861Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021514.xml (deflated 44%)
2022-11-23T02:20:52.4069261Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021518.xml (deflated 44%)
2022-11-23T02:20:52.4069676Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021522.xml (deflated 44%)
2022-11-23T02:20:52.4070131Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021527.xml (deflated 43%)
2022-11-23T02:20:52.4070553Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021531.xml (deflated 43%)
2022-11-23T02:20:52.4070963Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021536.xml (deflated 43%)
2022-11-23T02:20:52.4071371Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021540.xml (deflated 44%)
2022-11-23T02:20:52.4071796Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021544.xml (deflated 43%)
2022-11-23T02:20:52.4072212Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021549.xml (deflated 44%)
2022-11-23T02:20:52.4072626Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021554.xml (deflated 44%)
2022-11-23T02:20:52.4073051Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021558.xml (deflated 44%)
2022-11-23T02:20:52.4073466Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021603.xml (deflated 44%)
2022-11-23T02:20:52.4073880Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021608.xml (deflated 44%)
2022-11-23T02:20:52.4074286Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021613.xml (deflated 44%)
2022-11-23T02:20:52.4074703Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021618.xml (deflated 44%)
2022-11-23T02:20:52.4075116Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021622.xml (deflated 44%)
2022-11-23T02:20:52.4075534Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021627.xml (deflated 43%)
2022-11-23T02:20:52.4075950Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021631.xml (deflated 44%)
2022-11-23T02:20:52.4076359Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021636.xml (deflated 44%)
2022-11-23T02:20:52.4076775Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021641.xml (deflated 44%)
2022-11-23T02:20:52.4077212Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021645.xml (deflated 44%)
2022-11-23T02:20:52.4077632Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021650.xml (deflated 44%)
2022-11-23T02:20:52.4078046Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021655.xml (deflated 43%)
2022-11-23T02:20:52.4078455Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021700.xml (deflated 44%)
2022-11-23T02:20:52.4078890Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021704.xml (deflated 44%)
2022-11-23T02:20:52.4079288Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021709.xml (deflated 44%)
2022-11-23T02:20:52.4079700Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021713.xml (deflated 44%)
2022-11-23T02:20:52.4080122Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20221123021718.xml (deflated 44%)
2022-11-23T02:20:52.4080515Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021723.xml (deflated 43%)
2022-11-23T02:20:52.4080905Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021727.xml (deflated 43%)
2022-11-23T02:20:52.4081292Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021731.xml (deflated 43%)
2022-11-23T02:20:52.4081680Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021736.xml (deflated 43%)
2022-11-23T02:20:52.4082063Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021738.xml (deflated 43%)
2022-11-23T02:20:52.4082442Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021742.xml (deflated 44%)
2022-11-23T02:20:52.4082829Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021746.xml (deflated 43%)
2022-11-23T02:20:52.4083214Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021750.xml (deflated 43%)
2022-11-23T02:20:52.4083598Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021755.xml (deflated 43%)
2022-11-23T02:20:52.4083970Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021800.xml (deflated 43%)
2022-11-23T02:20:52.4084357Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021804.xml (deflated 43%)
2022-11-23T02:20:52.4084735Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021808.xml (deflated 43%)
2022-11-23T02:20:52.4085207Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021813.xml (deflated 43%)
2022-11-23T02:20:52.4085597Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021817.xml (deflated 43%)
2022-11-23T02:20:52.4086006Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021822.xml (deflated 43%)
2022-11-23T02:20:52.4086390Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021826.xml (deflated 43%)
2022-11-23T02:20:52.4086773Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021831.xml (deflated 43%)
2022-11-23T02:20:52.4087153Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021835.xml (deflated 43%)
2022-11-23T02:20:52.4087536Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021840.xml (deflated 43%)
2022-11-23T02:20:52.4087943Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021844.xml (deflated 43%)
2022-11-23T02:20:52.4088313Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021849.xml (deflated 43%)
2022-11-23T02:20:52.4088694Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021853.xml (deflated 43%)
2022-11-23T02:20:52.4089077Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021858.xml (deflated 44%)
2022-11-23T02:20:52.4089456Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021902.xml (deflated 43%)
2022-11-23T02:20:52.4089837Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021909.xml (deflated 43%)
2022-11-23T02:20:52.4090217Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021915.xml (deflated 43%)
2022-11-23T02:20:52.4090600Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021932.xml (deflated 43%)
2022-11-23T02:20:52.4090979Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021937.xml (deflated 44%)
2022-11-23T02:20:52.4091361Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021941.xml (deflated 43%)
2022-11-23T02:20:52.4091736Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021946.xml (deflated 44%)
2022-11-23T02:20:52.4092116Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021950.xml (deflated 43%)
2022-11-23T02:20:52.4092497Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021955.xml (deflated 43%)
2022-11-23T02:20:52.4092861Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123021959.xml (deflated 43%)
2022-11-23T02:20:52.4093242Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022004.xml (deflated 43%)
2022-11-23T02:20:52.4093623Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022008.xml (deflated 44%)
2022-11-23T02:20:52.4094007Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022012.xml (deflated 44%)
2022-11-23T02:20:52.4094385Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022017.xml (deflated 43%)
2022-11-23T02:20:52.4094787Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022021.xml (deflated 43%)
2022-11-23T02:20:52.4095393Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20221123022026.xml (deflated 43%)
2022-11-23T02:20:52.4095808Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022030.xml (deflated 44%)
2022-11-23T02:20:52.4096218Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022034.xml (deflated 43%)
2022-11-23T02:20:52.4096676Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20221123022038.xml (deflated 43%)
2022-11-23T02:20:52.4096884Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-11-23T02:20:52.4113018Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:20:52.4113158Z # Remove any previous test reports if they exist
2022-11-23T02:20:52.4113247Z rm -f usage-log-*.zip
2022-11-23T02:20:52.4113393Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T02:20:52.4113496Z # so check to see if the file exists first
2022-11-23T02:20:52.4113587Z if [ -f 'usage_log.txt' ]; then
2022-11-23T02:20:52.4113707Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T02:20:52.4113769Z fi
2022-11-23T02:20:52.4124741Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:20:52.4124806Z env:
2022-11-23T02:20:52.4124958Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:52.4125119Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:52.4125257Z   FILE_SUFFIX: test-distributed-1-2-linux.2xlarge_9655066953
2022-11-23T02:20:52.4125334Z ##[endgroup]
2022-11-23T02:20:52.4687515Z   adding: usage_log.txt (deflated 94%)
2022-11-23T02:20:52.4722776Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:20:52.4722866Z with:
2022-11-23T02:20:52.4723030Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T02:20:52.4723146Z   retention-days: 14
2022-11-23T02:20:52.4723234Z   if-no-files-found: warn
2022-11-23T02:20:52.4723321Z   path: test-jsons-*.zip
2022-11-23T02:20:52.4723389Z   name: artifact
2022-11-23T02:20:52.4723469Z   s3-bucket: gha-artifacts
2022-11-23T02:20:52.4723541Z   region: us-east-1
2022-11-23T02:20:52.4723608Z env:
2022-11-23T02:20:52.4723679Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:52.4723831Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:52.4723900Z ##[endgroup]
2022-11-23T02:20:52.8619976Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:20:52.8620462Z With the provided path, there will be 1 file uploaded
2022-11-23T02:20:52.8620933Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T02:20:52.8677996Z Starting upload of test-jsons-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:53.0444721Z Finished upload of test-jsons-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:53.0583665Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:20:53.0583884Z with:
2022-11-23T02:20:53.0584078Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T02:20:53.0584296Z   retention-days: 14
2022-11-23T02:20:53.0584492Z   if-no-files-found: error
2022-11-23T02:20:53.0584690Z   path: test-reports-*.zip
2022-11-23T02:20:53.0584876Z   name: artifact
2022-11-23T02:20:53.0585055Z   s3-bucket: gha-artifacts
2022-11-23T02:20:53.0585228Z   region: us-east-1
2022-11-23T02:20:53.0585392Z env:
2022-11-23T02:20:53.0585559Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:53.0585811Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:53.0586138Z ##[endgroup]
2022-11-23T02:20:53.3983362Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:20:53.3984005Z With the provided path, there will be 1 file uploaded
2022-11-23T02:20:53.3984538Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T02:20:53.3991368Z Starting upload of test-reports-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:53.7000740Z Finished upload of test-reports-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:53.7136077Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:20:53.7136298Z with:
2022-11-23T02:20:53.7136571Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T02:20:53.7136787Z   retention-days: 14
2022-11-23T02:20:53.7136982Z   if-no-files-found: ignore
2022-11-23T02:20:53.7137166Z   path: usage-log-*.zip
2022-11-23T02:20:53.7137346Z   name: artifact
2022-11-23T02:20:53.7137522Z   s3-bucket: gha-artifacts
2022-11-23T02:20:53.7137695Z   region: us-east-1
2022-11-23T02:20:53.7137864Z env:
2022-11-23T02:20:53.7138031Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:53.7138282Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:53.7138528Z ##[endgroup]
2022-11-23T02:20:54.0494447Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:20:54.0495462Z With the provided path, there will be 1 file uploaded
2022-11-23T02:20:54.0495992Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T02:20:54.0502631Z Starting upload of usage-log-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:54.2268793Z Finished upload of usage-log-test-distributed-1-2-linux.2xlarge_9655066953.zip
2022-11-23T02:20:54.2399756Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T02:20:54.2400007Z # shellcheck disable=SC2156
2022-11-23T02:20:54.2400306Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T02:20:54.2411445Z shell: /usr/bin/bash -e {0}
2022-11-23T02:20:54.2411632Z env:
2022-11-23T02:20:54.2411796Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:54.2412064Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:54.2412314Z ##[endgroup]
2022-11-23T02:20:55.8667338Z ##[group]Run set -x
2022-11-23T02:20:55.8667542Z set -x
2022-11-23T02:20:55.8667749Z python3 -m pip install -r requirements.txt
2022-11-23T02:20:55.8667999Z python3 -m pip install boto3==1.19.12
2022-11-23T02:20:55.8668293Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:20:55.8679341Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:20:55.8679560Z env:
2022-11-23T02:20:55.8679736Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:20:55.8680010Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:20:55.8680267Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T02:20:55.8680458Z   BRANCH: master
2022-11-23T02:20:55.8680639Z   TEST_CONFIG: distributed
2022-11-23T02:20:55.8680809Z   SHARD_NUMBER: 1
2022-11-23T02:20:55.8681022Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-11-23T02:20:55.8681229Z   PR_NUMBER: 
2022-11-23T02:20:55.8681430Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T02:20:55.8681638Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T02:20:55.8681846Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:20:55.8682039Z   TAG: 
2022-11-23T02:20:55.8682202Z   WORKFLOW_ID: 3528293562
2022-11-23T02:20:55.8682513Z   GITHUB_TOKEN: ***
2022-11-23T02:20:55.8682706Z   GHA_WORKFLOW_JOB_ID: 9655066953
2022-11-23T02:20:55.8682892Z ##[endgroup]
2022-11-23T02:20:55.8727421Z + python3 -m pip install -r requirements.txt
2022-11-23T02:20:56.0908609Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:20:56.1191598Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-11-23T02:20:56.1219193Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2022-11-23T02:20:56.1227421Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-11-23T02:20:56.1235767Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.0)
2022-11-23T02:20:56.1598001Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-11-23T02:20:56.1607258Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T02:20:56.1682703Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-11-23T02:20:56.1690675Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T02:20:56.1908877Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T02:20:56.2074388Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-11-23T02:20:56.2082886Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-11-23T02:20:56.2088733Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2022-11-23T02:20:56.2098759Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-11-23T02:20:56.2117057Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.8.0)
2022-11-23T02:20:56.2188155Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2022-11-23T02:20:56.2348164Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2022-11-23T02:20:56.2371019Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2022-11-23T02:20:56.2388441Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-11-23T02:20:56.2637944Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-11-23T02:20:56.2648568Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4)
2022-11-23T02:20:56.2665132Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T02:20:56.2677015Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T02:20:56.2697419Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T02:20:56.2705829Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T02:20:56.2856235Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-11-23T02:20:56.2909883Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2022-11-23T02:20:56.3491517Z + python3 -m pip install boto3==1.19.12
2022-11-23T02:20:56.5570108Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:20:56.5751842Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-11-23T02:20:56.5801110Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-11-23T02:20:56.5843026Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-11-23T02:20:56.5864940Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-11-23T02:20:56.5889367Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-11-23T02:20:56.5909150Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T02:20:56.6062026Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T02:20:56.7876868Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:21:04.5003959Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T02:21:04.5004284Z 
2022-11-23T02:21:04.5004653Z ----- Historic stats comparison result ------
2022-11-23T02:21:04.5004944Z 
2022-11-23T02:21:04.5005104Z     job: linux-focal-py3.7-gcc7
2022-11-23T02:21:04.5005331Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:21:04.5005473Z 
2022-11-23T02:21:04.5005841Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T02:21:04.5006021Z 
2022-11-23T02:21:04.5006090Z     : (master)
2022-11-23T02:21:04.5006241Z     |
2022-11-23T02:21:04.5006435Z     * 1cfd3858ac (HEAD)              total time  2036.34s
2022-11-23T02:21:04.5008510Z     * 26322544b8 (base)   6 reports, total time  3670.61s ± 2959.87s
2022-11-23T02:21:04.5008881Z     * 7f4b4d2827          6 reports, total time  3666.21s ± 2982.42s
2022-11-23T02:21:04.5009196Z     * b50699f247          6 reports, total time  3707.01s ± 3010.86s
2022-11-23T02:21:04.5009509Z     * 8bf8e4d71e          6 reports, total time  3679.02s ± 2959.94s
2022-11-23T02:21:04.5009825Z     * ce856cee7e          6 reports, total time  3732.77s ± 3067.45s
2022-11-23T02:21:04.5010123Z     * 391b593ca2          6 reports, total time  3766.83s ± 3183.26s
2022-11-23T02:21:04.5010434Z     * 5bba783d21          6 reports, total time  3688.41s ± 3016.52s
2022-11-23T02:21:04.5010788Z     * ea920a1115          6 reports, total time  3713.47s ± 3039.67s
2022-11-23T02:21:04.5011085Z     * 74e62a1fef          6 reports, total time  3657.38s ± 2942.16s
2022-11-23T02:21:04.5011396Z     * 00b7d8ef23          6 reports, total time  3735.20s ± 2985.62s
2022-11-23T02:21:04.5011595Z     |
2022-11-23T02:21:04.5011732Z     :
2022-11-23T02:21:04.5011832Z 
2022-11-23T02:21:04.5011951Z Removed  (across  894 suites)     0 tests, totaling     0.00s
2022-11-23T02:21:04.5012210Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T02:21:04.5012467Z Added    (across  102 suites)  1325 tests, totaling +2436.74s
2022-11-23T02:21:04.5572726Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T02:21:04.5572994Z with:
2022-11-23T02:21:04.5573230Z env:
2022-11-23T02:21:04.5573452Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:21:04.5573737Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:21:04.5574051Z ##[endgroup]
2022-11-23T02:21:04.5603581Z ##[group]Run set -eou pipefail
2022-11-23T02:21:04.5603790Z set -eou pipefail
2022-11-23T02:21:04.5603969Z 
2022-11-23T02:21:04.5604202Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T02:21:04.5604451Z for _ in $(seq 1440); do
2022-11-23T02:21:04.5604734Z     # Break if no ssh session exists anymore
2022-11-23T02:21:04.5605034Z     if [ "$(who)" = "" ]; then
2022-11-23T02:21:04.5605216Z       break
2022-11-23T02:21:04.5605367Z     fi
2022-11-23T02:21:04.5605566Z     echo "."
2022-11-23T02:21:04.5605738Z     sleep 5
2022-11-23T02:21:04.5605893Z done
2022-11-23T02:21:04.5617389Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:21:04.5617609Z env:
2022-11-23T02:21:04.5617771Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:21:04.5618040Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:21:04.5618289Z ##[endgroup]
2022-11-23T02:21:04.5642051Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T02:21:04.5725715Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:21:04.5726031Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:21:04.5726287Z # shellcheck disable=SC2046
2022-11-23T02:21:04.5726515Z docker stop $(docker ps -q) || true
2022-11-23T02:21:04.5726727Z # Prune all of the docker images
2022-11-23T02:21:04.5726939Z docker system prune -af
2022-11-23T02:21:04.5737166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:21:04.5737372Z env:
2022-11-23T02:21:04.5737545Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:21:04.5737814Z   DOCKER_CONTAINER_ID: 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:21:04.5738050Z ##[endgroup]
2022-11-23T02:21:05.0583092Z 5347e8488c91
2022-11-23T02:21:06.6653212Z Deleted Containers:
2022-11-23T02:21:06.6653778Z 5347e8488c914580ead0c9ccd404898dfd06c323f93e604b4650d98e2a4f6bd5
2022-11-23T02:21:06.6653990Z 
2022-11-23T02:21:13.1722702Z Deleted Images:
2022-11-23T02:21:13.1723331Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T02:21:13.1724038Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:128a8060bc8d0d2006ae18b574265c3aa935b731c43947b53188b825fa718d6b
2022-11-23T02:21:13.1724491Z deleted: sha256:ac950c018cbcc41d6d7a73f33ac11fc7d5ae0cb1e9dbf1f471e7802e3c2787b3
2022-11-23T02:21:13.1724836Z deleted: sha256:b2992f8a5deb11d65e86729b600a9462d189b6a0e2d64ddf46a935b534edee73
2022-11-23T02:21:13.1725221Z deleted: sha256:377952e5d625f33006da79581ca002fe8708e712667cd20f83e3be62d7468554
2022-11-23T02:21:13.1725511Z deleted: sha256:9843138b6a57d457beab855e3460284d1866ba91a97f30314c279e5d91fe8d13
2022-11-23T02:21:13.1725823Z deleted: sha256:ef7ffe0d473b250807d16fbe59333f43b284836e2f113fc021b51c63b2e011e7
2022-11-23T02:21:13.1726149Z deleted: sha256:36e6aaa83981278ab4c62090dd61d8e6b9646d6d29d3e8cf7fe3fa5d9513cf0a
2022-11-23T02:21:13.1726454Z deleted: sha256:547328983788be2665bb6e828066d6feeee1e5b15bf2ddc6bf2f259552437c2a
2022-11-23T02:21:13.1726759Z deleted: sha256:e2e21028b051073a8bc6980590edf442bdc3e1306e0028b49dfd55cf2c91b9ee
2022-11-23T02:21:13.1727081Z deleted: sha256:43b659b86bc116dbce677f866a791ec472a054d893eceadb868a83fc10ddf981
2022-11-23T02:21:13.1727593Z deleted: sha256:aee61630fbc682636784477102a9b6b79fe2d838f21e59d50668d685fdb299f8
2022-11-23T02:21:13.1727899Z deleted: sha256:dc980582b7d0ab6953bd378bfa7248ef8e950d4463bf46788b8c9fdd38de09c6
2022-11-23T02:21:13.1728225Z deleted: sha256:bc41e9d5653f1c334e1531d0b8f6a3c2c71eb1b5faa6ccf4f1982e341f3f51b7
2022-11-23T02:21:13.1728558Z deleted: sha256:cc9cae77052e164c6a73da731c9b62ff3d69b8ab2ec5e497ad84e27f512755a3
2022-11-23T02:21:13.1728870Z deleted: sha256:90bb77a23d716774ba1e5e2088208bd8737f96f2bbfb9a08e784f7251c6f76d6
2022-11-23T02:21:13.1729177Z deleted: sha256:8964a3bf610c6b0f62ab78a367344a7dd597b5c72219923f1e09001e2e3b79f3
2022-11-23T02:21:13.1729477Z deleted: sha256:b75210d5af1517b607b80542cb9a815c9703720427a8b559e399cb7a2a7ba8a7
2022-11-23T02:21:13.1729789Z deleted: sha256:14f0bcbce4ae20f68c2e993ad2746d59f4f456c783d5862b894e7245ff2940f4
2022-11-23T02:21:13.1730169Z deleted: sha256:9b6bcbca6fcb1af72a85da5f16b8295e3c678611151b4d16f9a1575fe3db8492
2022-11-23T02:21:13.1730502Z deleted: sha256:f199e2cb9461ed2044209904fafeb4de59b7c08fea775715b633eaaa1987d76a
2022-11-23T02:21:13.1730810Z deleted: sha256:70b4e3728651e26c75a4845e785a5e2e91e3a7d3203833638a093f2e1d8b221a
2022-11-23T02:21:13.1731097Z deleted: sha256:1d5959fe962bc85cd740541c582b145ce1374b02b323beff68a0426d63c86a49
2022-11-23T02:21:13.1731398Z deleted: sha256:87062382eaa8d09a8a232a7d4586e0241eb319eb75654d7d652183e890363f0a
2022-11-23T02:21:13.1731690Z deleted: sha256:31666c108535eb4050261a140e3f31726c89b938120e1e39ff1d81c8cab6a349
2022-11-23T02:21:13.1731981Z deleted: sha256:98a161950ac2230e581a5e15e4905b1855e6ee1fe94558d8473e2e65e375475f
2022-11-23T02:21:13.1732288Z deleted: sha256:c4276d6adb8f7a22a7dbacdc0f6552113ccc525a19608c9f201399a95862928c
2022-11-23T02:21:13.1732610Z deleted: sha256:61a0274c3c6c91a0cd553a026ed8b8f1a924c35295bb41c32927f7e6e4660d39
2022-11-23T02:21:13.1732935Z deleted: sha256:57f2baf11e15f4b61fd2de15d0f7c10150e83aca1de166f644c472b2780428e7
2022-11-23T02:21:13.1733241Z deleted: sha256:2a5723d4a8c0331709b4ff22a4d78f12e5e1396a281832c1c7f1ded6257c06b6
2022-11-23T02:21:13.1733548Z deleted: sha256:b235c0c7e72494541107ca95d5d5445eb1e94fbd4ce40b73d7d98010105de22a
2022-11-23T02:21:13.1733848Z deleted: sha256:74e83991baa883fd71a6668d703f0419506e79c602d19f6888efc9ee60eda5a0
2022-11-23T02:21:13.1734165Z deleted: sha256:149935fef489f40eadd6a9475abd8bd67700c1bc9fdff273fdfcc31ef4baa465
2022-11-23T02:21:13.1734488Z deleted: sha256:635c2c6690b7bbedfda868b6fb88cc966a2c005b9540839fa78c30bf0e8d0e08
2022-11-23T02:21:13.1734816Z deleted: sha256:c962aa8089ba331791c2b2b83740f4db5b1aac8ef8e891cdc595c3fdb5310dbc
2022-11-23T02:21:13.1735478Z deleted: sha256:b51083cce1d6de548b12fd7b666205f728d375ce6124109e79212b44fb198d87
2022-11-23T02:21:13.1735791Z deleted: sha256:e3c257e1d8787783e8f79a25fcd497cce01bd7bedcf407b8eae290d81428f1ca
2022-11-23T02:21:13.1736114Z deleted: sha256:68698630dd71c4db0eacdac941d296ab20d7c4370c8122793a2d452f0c6f476d
2022-11-23T02:21:13.1736431Z deleted: sha256:aca108602e826c17f32620b9ade1562004117a689df4bb1b4ae46b638a36c2a1
2022-11-23T02:21:13.1736734Z deleted: sha256:93b007516b03c7ea1975114590b8e43ae2e9309180acc0ee20b586dc41ff0e4c
2022-11-23T02:21:13.1737032Z deleted: sha256:4b0b6cc64e7afa0bebb597b5bb5755852430b284745950b748fdf945ee7c7343
2022-11-23T02:21:13.1737334Z deleted: sha256:510466f40335397d70041861d895d661aba31b0f89fe9ed823b502709e6a048c
2022-11-23T02:21:13.1737639Z deleted: sha256:a940b4fb25d752aed6e2effcfdd74338042ad2d2f1536438dc53a065b609d3b1
2022-11-23T02:21:13.1737952Z deleted: sha256:effe44ab85505068890636f69282d6246ca2c13d558b18d5ff52d44acedf52e4
2022-11-23T02:21:13.1738277Z deleted: sha256:2b26ee6f3f7e8b943ac9ae9d70b6fff3e4ebd4b5c42d530a5939aab7a5a8e968
2022-11-23T02:21:13.1738611Z deleted: sha256:b86b5801635d9fb03bccba09570b1c9c770d5eb6e6d1a1efde30df10444cbcda
2022-11-23T02:21:13.1738922Z deleted: sha256:491df340aee36942648573aeef75d3f1ece9f65d74f7d6a025b41ac68b17774b
2022-11-23T02:21:13.1739236Z deleted: sha256:e10409ac2ee50fb36987bc7932b127299cffc29888a5917ba29c36024f6fd3b6
2022-11-23T02:21:13.1739562Z deleted: sha256:eecb37d5ef051b777dce74f18fff350cc8c9c687f178d4321a1f4ba3a66f8e17
2022-11-23T02:21:13.1739970Z deleted: sha256:c033ce386c9769757faf2693533fea2abcd3e7c50f538aed3d0202907220131a
2022-11-23T02:21:13.1740278Z deleted: sha256:01cef2346a033872b4dc7c3938eb6be98e8c9b54ec05d98b84081c122f21d0e5
2022-11-23T02:21:13.1740596Z deleted: sha256:743ac6a4351c56c1d41bf297ddb1978e237e87ce4f78ef5eae80927c64b73c6e
2022-11-23T02:21:13.1740909Z deleted: sha256:6ae19417bc8a33832e82e9051aeca485e2839a197ed86584713056b959d53f6e
2022-11-23T02:21:13.1741217Z deleted: sha256:bd69c9e420df78782dec5b44ac56d41ee0fcb2deb00d6b2b439e5a33b79eb5d9
2022-11-23T02:21:13.1741542Z deleted: sha256:26986e9c9bf97e7a71da10396a6069f1d61d84cffa373ef9295ae547857faf74
2022-11-23T02:21:13.1742046Z deleted: sha256:a2fade5710ceea279a50353534afd459a814b185fe0f4a2baa00ed296a7e73a9
2022-11-23T02:21:13.1742423Z deleted: sha256:07108459e4c8e804d5f79b0a75a3902a681d1d77f1651dac39c5944cc9070846
2022-11-23T02:21:13.1742782Z deleted: sha256:afde38d23fe01a4fe20b0c36952db88f94e3ae5f90196a1f0da7f668ced05a1a
2022-11-23T02:21:13.1743119Z deleted: sha256:329b465bb32a9dc01247f25c90efd8ce5a2d1ca603c32f17c6becbf911de8ea9
2022-11-23T02:21:13.1743502Z deleted: sha256:2bf3e548b369a3c5003bf543698d18f933ece97bc5bcfd8bb7454dd4f3c02ec1
2022-11-23T02:21:13.1743863Z deleted: sha256:f4462d5b2da2985f37409c9b257afd2b9fb82356ce4e43e804ee34214242e34a
2022-11-23T02:21:13.1744107Z 
2022-11-23T02:21:13.1744499Z Total reclaimed space: 9.776GB
2022-11-23T02:21:13.1811300Z Post job cleanup.
2022-11-23T02:21:13.1846906Z Post job cleanup.
2022-11-23T02:21:13.2939017Z [command]/usr/bin/git version
2022-11-23T02:21:13.2983898Z git version 2.37.1
2022-11-23T02:21:13.3022105Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d4e95346-826b-4be4-84b5-80b3e528f9aa' before making global git config changes
2022-11-23T02:21:13.3022742Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T02:21:13.3026507Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T02:21:13.3072202Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T02:21:13.3102413Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T02:21:13.3502488Z Entering 'android/libs/fbjni'
2022-11-23T02:21:13.3543440Z Entering 'third_party/FP16'
2022-11-23T02:21:13.3588624Z Entering 'third_party/FXdiv'
2022-11-23T02:21:13.3627296Z Entering 'third_party/NNPACK'
2022-11-23T02:21:13.3673942Z Entering 'third_party/QNNPACK'
2022-11-23T02:21:13.3715134Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T02:21:13.3748860Z Entering 'third_party/XNNPACK'
2022-11-23T02:21:13.3793885Z Entering 'third_party/benchmark'
2022-11-23T02:21:13.3827835Z Entering 'third_party/cpuinfo'
2022-11-23T02:21:13.3862077Z Entering 'third_party/cub'
2022-11-23T02:21:13.3896851Z Entering 'third_party/cudnn_frontend'
2022-11-23T02:21:13.3935267Z Entering 'third_party/cutlass'
2022-11-23T02:21:13.3975887Z Entering 'third_party/eigen'
2022-11-23T02:21:13.4012279Z Entering 'third_party/fbgemm'
2022-11-23T02:21:13.4045956Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T02:21:13.4079589Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T02:21:13.4112234Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T02:21:13.4146143Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T02:21:13.4180670Z Entering 'third_party/flatbuffers'
2022-11-23T02:21:13.4217714Z Entering 'third_party/fmt'
2022-11-23T02:21:13.4251374Z Entering 'third_party/foxi'
2022-11-23T02:21:13.4285090Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T02:21:13.4323472Z Entering 'third_party/gloo'
2022-11-23T02:21:13.4357840Z Entering 'third_party/googletest'
2022-11-23T02:21:13.4391627Z Entering 'third_party/ideep'
2022-11-23T02:21:13.4424128Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T02:21:13.4459804Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T02:21:13.4499895Z Entering 'third_party/ios-cmake'
2022-11-23T02:21:13.4533766Z Entering 'third_party/ittapi'
2022-11-23T02:21:13.4568283Z Entering 'third_party/kineto'
2022-11-23T02:21:13.4602387Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T02:21:13.4636251Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T02:21:13.4671662Z Entering 'third_party/nccl/nccl'
2022-11-23T02:21:13.4705857Z Entering 'third_party/neon2sse'
2022-11-23T02:21:13.4740200Z Entering 'third_party/nlohmann'
2022-11-23T02:21:13.4774401Z Entering 'third_party/onnx'
2022-11-23T02:21:13.4819393Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T02:21:13.4852725Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T02:21:13.4889153Z Entering 'third_party/onnx-tensorrt'
2022-11-23T02:21:13.4922283Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T02:21:13.4962197Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T02:21:13.4996935Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T02:21:13.5031914Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T02:21:13.5070938Z Entering 'third_party/pocketfft'
2022-11-23T02:21:13.5105892Z Entering 'third_party/protobuf'
2022-11-23T02:21:13.5144854Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T02:21:13.5178254Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T02:21:13.5214004Z Entering 'third_party/psimd'
2022-11-23T02:21:13.5247969Z Entering 'third_party/pthreadpool'
2022-11-23T02:21:13.5282047Z Entering 'third_party/pybind11'
2022-11-23T02:21:13.5315701Z Entering 'third_party/python-enum'
2022-11-23T02:21:13.5349430Z Entering 'third_party/python-peachpy'
2022-11-23T02:21:13.5382955Z Entering 'third_party/python-six'
2022-11-23T02:21:13.5417047Z Entering 'third_party/sleef'
2022-11-23T02:21:13.5450690Z Entering 'third_party/tbb'
2022-11-23T02:21:13.5485223Z Entering 'third_party/tensorpipe'
2022-11-23T02:21:13.5519798Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T02:21:13.5553528Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T02:21:13.5586223Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T02:21:13.5619689Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T02:21:13.5652067Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T02:21:13.5688462Z Entering 'third_party/zstd'
2022-11-23T02:21:13.5735914Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T02:21:13.5761519Z http.https://github.com/.extraheader
2022-11-23T02:21:13.5769064Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T02:21:13.5798878Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T02:21:13.6052962Z Entering 'android/libs/fbjni'
2022-11-23T02:21:13.6072957Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6099619Z Entering 'third_party/FP16'
2022-11-23T02:21:13.6120222Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6145880Z Entering 'third_party/FXdiv'
2022-11-23T02:21:13.6166463Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6193591Z Entering 'third_party/NNPACK'
2022-11-23T02:21:13.6212911Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6239356Z Entering 'third_party/QNNPACK'
2022-11-23T02:21:13.6259708Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6286675Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T02:21:13.6306730Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6333941Z Entering 'third_party/XNNPACK'
2022-11-23T02:21:13.6354315Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6390532Z Entering 'third_party/benchmark'
2022-11-23T02:21:13.6409938Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6436126Z Entering 'third_party/cpuinfo'
2022-11-23T02:21:13.6456916Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6491902Z Entering 'third_party/cub'
2022-11-23T02:21:13.6511635Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6537983Z Entering 'third_party/cudnn_frontend'
2022-11-23T02:21:13.6557889Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6589827Z Entering 'third_party/cutlass'
2022-11-23T02:21:13.6608984Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6641956Z Entering 'third_party/eigen'
2022-11-23T02:21:13.6662357Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6691495Z Entering 'third_party/fbgemm'
2022-11-23T02:21:13.6712022Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6749042Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T02:21:13.6767750Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6794087Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T02:21:13.6814776Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6840886Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T02:21:13.6861575Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6887879Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T02:21:13.6907049Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6934743Z Entering 'third_party/flatbuffers'
2022-11-23T02:21:13.6954553Z http.https://github.com/.extraheader
2022-11-23T02:21:13.6984371Z Entering 'third_party/fmt'
2022-11-23T02:21:13.7003897Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7029794Z Entering 'third_party/foxi'
2022-11-23T02:21:13.7049579Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7075486Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T02:21:13.7096303Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7133081Z Entering 'third_party/gloo'
2022-11-23T02:21:13.7153039Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7180097Z Entering 'third_party/googletest'
2022-11-23T02:21:13.7200192Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7226178Z Entering 'third_party/ideep'
2022-11-23T02:21:13.7246345Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7271988Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T02:21:13.7291275Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7319853Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T02:21:13.7339663Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7372836Z Entering 'third_party/ios-cmake'
2022-11-23T02:21:13.7393540Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7419757Z Entering 'third_party/ittapi'
2022-11-23T02:21:13.7439825Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7466033Z Entering 'third_party/kineto'
2022-11-23T02:21:13.7486293Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7513429Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T02:21:13.7533474Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7560100Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T02:21:13.7580786Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7608440Z Entering 'third_party/nccl/nccl'
2022-11-23T02:21:13.7628389Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7654742Z Entering 'third_party/neon2sse'
2022-11-23T02:21:13.7675182Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7701730Z Entering 'third_party/nlohmann'
2022-11-23T02:21:13.7722035Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7759699Z Entering 'third_party/onnx'
2022-11-23T02:21:13.7779848Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7817051Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T02:21:13.7836256Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7871476Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T02:21:13.7890939Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7927903Z Entering 'third_party/onnx-tensorrt'
2022-11-23T02:21:13.7947748Z http.https://github.com/.extraheader
2022-11-23T02:21:13.7983207Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T02:21:13.8002866Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8041164Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T02:21:13.8060785Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8086461Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T02:21:13.8106222Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8131813Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T02:21:13.8151354Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8181523Z Entering 'third_party/pocketfft'
2022-11-23T02:21:13.8201219Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8227029Z Entering 'third_party/protobuf'
2022-11-23T02:21:13.8247552Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8277096Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T02:21:13.8297034Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8322662Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T02:21:13.8342299Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8369849Z Entering 'third_party/psimd'
2022-11-23T02:21:13.8390397Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8415964Z Entering 'third_party/pthreadpool'
2022-11-23T02:21:13.8435457Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8461782Z Entering 'third_party/pybind11'
2022-11-23T02:21:13.8480880Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8506690Z Entering 'third_party/python-enum'
2022-11-23T02:21:13.8526743Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8552405Z Entering 'third_party/python-peachpy'
2022-11-23T02:21:13.8572423Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8599606Z Entering 'third_party/python-six'
2022-11-23T02:21:13.8619556Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8645259Z Entering 'third_party/sleef'
2022-11-23T02:21:13.8664574Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8691675Z Entering 'third_party/tbb'
2022-11-23T02:21:13.8711497Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8740926Z Entering 'third_party/tensorpipe'
2022-11-23T02:21:13.8760781Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8787251Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T02:21:13.8806828Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8833299Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T02:21:13.8852985Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8878745Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T02:21:13.8899564Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8926277Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T02:21:13.8945508Z http.https://github.com/.extraheader
2022-11-23T02:21:13.8970975Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T02:21:13.8990266Z http.https://github.com/.extraheader
2022-11-23T02:21:13.9019261Z Entering 'third_party/zstd'
2022-11-23T02:21:13.9038808Z http.https://github.com/.extraheader
2022-11-23T02:21:13.9273571Z Cleaning up orphan processes